[{"content":"","date":"31 May 2026","externalUrl":null,"permalink":"/tags/backups/","section":"Tags","summary":"","title":"Backups","type":"tags"},{"content":"","date":"31 May 2026","externalUrl":null,"permalink":"/categories/","section":"Categories","summary":"","title":"Categories","type":"categories"},{"content":"","date":"31 May 2026","externalUrl":null,"permalink":"/categories/homelab/","section":"Categories","summary":"","title":"Homelab","type":"categories"},{"content":"","date":"31 May 2026","externalUrl":null,"permalink":"/tags/homelab/","section":"Tags","summary":"","title":"Homelab","type":"tags"},{"content":"","date":"31 May 2026","externalUrl":null,"permalink":"/homelab/","section":"Homelabs","summary":"","title":"Homelabs","type":"homelab"},{"content":"","date":"31 May 2026","externalUrl":null,"permalink":"/","section":"Josiah","summary":"","title":"Josiah","type":"page"},{"content":"","date":"31 May 2026","externalUrl":null,"permalink":"/tags/mistakes/","section":"Tags","summary":"","title":"Mistakes","type":"tags"},{"content":"","date":"31 May 2026","externalUrl":null,"permalink":"/tags/","section":"Tags","summary":"","title":"Tags","type":"tags"},{"content":"The homelab is intended to be a place to make mistakes, and this is good because I make my fair share number of them. Today\u0026rsquo;s mistakes involved inconsistant databases, ChatGPT, and (thankfully!) usefull backups to save the day! I\u0026rsquo;ll walk you through the incident the same way I experienced it.\nIt began as I was working to deploy a new service (yes, ANOTHER one), this time a bookmarking app called Karakeep. It has OIDC suport, so I went to Authentik to setup the connection. I was intially met with the normal login screen, but after I typed in my username and password I was then met with a generic \u0026ldquo;Server Error\u0026rdquo; message. \u0026ldquo;That\u0026rsquo;s odd\u0026rdquo; I muttered as I tried logging in through a private window and with a different account. All three attemts resulting in the same \u0026ldquo;Server Error\u0026rdquo; message.\nSo, I checked the logs and recieved this\u0026hellip;\nI had\u0026hellip;no idea what I was looking at or why anything like this would have come up. So I threw it into ChatGPT and asked it what the heck is going on here.\nChatGPT respoded by pointing to the first line (relation: \u0026quot;authentik_core_user_ak_groups\u0026quot; does not exist) explaning that a \u0026lsquo;relation\u0026rsquo; in PostgreSQL is basically a table. Authentik is trying to query the table authentik_core_user_ak_groups and PostgreSQL is responding saying it doesn\u0026rsquo;t exist. This results in a fatal error because that table should exist. ChatGPT hypothosized that this was due to a failed or incomplete upgrade where some database migrations didn\u0026rsquo;t happen properly.\nBased on this diagnosis, I knew that I was well out of my depth. Though I know some things, I am not a Database Administrator, so I decied to let ChatGPT guide me through the most likely fixes. I know enough about enough that I could spot whether a command that ChatGPT was going to ask me to run would \u0026lsquo;accidentally\u0026rsquo; delete all my data.\nIt was at this point that I rememebred that I had started upgrading Authentik about a week or so ago. I was a few versions behind and had a few minutes, but for some reason I only updated by one version. Maybe something went wrong durring that upgrade?\nChatGPT had me check various things trying to see if the database migrations failed or if some wern\u0026rsquo;t applied. Instead of rehashing it all, here is a link to my conversation so you can see in real time all the stuff that was tried.\nhttps://chatgpt.com/share/6a1c7f33-5b38-83ea-8e0b-78a3ca6b5465\nNear the end, it and I determined that all the migrations had happened successfully and completely. There were no inconsistantcies in the database or in Authentik\u0026rsquo;s code. Each were saying that everything was applied, but clearly they had different ideas about what \u0026ldquo;everything\u0026rdquo; means. Since that was the case the only option left was to roll back to an older version and restore from a backup. I went digging in my GitHub repo with all my encypted envronment files. To my surprise, the version that was curently installed and running (2025.8) was the same version that had been running for the past 3 months. But I had upgraded, haden\u0026rsquo;t I?\nIt was at this point that my brain decided to finally give me all the information from that fateful day. I had indeed upgraded to 2025.10. The database migrations were comeplete and successful. Then, for some unknown and rediculous reason, I decided I didn\u0026rsquo;t have enough time to upgrade all the way to the newest version (2026.something), so I just changed the version tag back to 2025.8 and re-deployed, leaving the database with all the changed made in the 2025.10 upgrade.\nSo, the 2025.8 code was looking for a table authentik_core_user_ak_groups that had been removed in 2025.10. This what was causing the inconsistancy. There was no failure, just two incompatable systems. As the title states, the database was from the future. (thanks to ChatGPT for that idea)\nThankfully, this happened only a week ago. I export all the databases for my apps every night and keep 30 days worth of these backups. The restore process was as simple as stopping all but the db container, removing the current database, and restoring the backup to a new database. Then I started up all the containers, and voila, I was able to login no problem and finally create the new provider for Karakeep!\nSo, lessons learned? Don\u0026rsquo;t rollback upgrades without actually rolling them back completely. Oh and also, if an upgrade is successful, leave it. Sounds obvious, but hey, we all have moments of un-clarity sometimes.\n(I still have no reason why I didn\u0026rsquo;t just leave it at 2025.10)\n","date":"31 May 2026","externalUrl":null,"permalink":"/homelab/authentik-rollback-disaster/","section":"Homelabs","summary":"","title":"The Database From the Future","type":"homelab"},{"content":"","date":"1 March 2026","externalUrl":null,"permalink":"/tags/aws/","section":"Tags","summary":"","title":"Aws","type":"tags"},{"content":"This is the second post on AWS IAM, and I\u0026rsquo;ll be talking about Roles, IAM Security Tools, and ending with some best practices.\nRoles # We saw in the last post that you can assign policies to user accounts to allow/deny certain actions within your AWS account. But what if you wanted a third party to perform some kind of action on your account? What about an automated system? Is there any way to limit their access, or do they just get full carte blanche access to your entire account?\nWell, of course you can limit their access! The feature is IAM Roles. These are essentially the same kind of security policies discussed in the last post, but applied to different identities or agents.\nFrom Amazon\u0026rsquo;s own documentation:\nA role is an IAM identity that you can create in your account that has specific permissions. An IAM role has some similarities to an IAM user. Roles and users are both AWS identities with permissions policies that determine what the identity can and cannot do in AWS. However, instead of being uniquely associated with one person, a role can be assumed by anyone who needs it. A role does not have standard long-term credentials such as a password or access keys associated with it. Instead, when you assume a role, it provides you with temporary security credentials for your role session.\nThis is huge for security in limiting the potential blast-radius. Roles allow you to give temporary permissions to user accounts, permissions that expire after a set amount of time. So the third-party contractor\u0026rsquo;s user dont just have free-reign to do as he pleases, he has to assume the role first, which is an action logged and auditable by CloudWatch. Your Lamda function\u0026rsquo;s user can assume a role to delete VMs, but that user account normally has no permissions otherwise. Even normal user accounts can use roles to limit their admin privileges to only when they need them, reducing any costly mistakes.\nIAM Credential Report # IAM Credential Report is an account-level security tool that can export the current status of all user accounts and their various credential into a CSV file. You can then edit and manipulate this file as you wish, to determine, for example, which users have MFA enabled, or which users have not logged in in the past 90 days. This report also has information regarding access keys and when they were last rotated, super important for maintaining a strong security posture.\nIAM Access Advisor # IAM Access Advisor (renamed recently to simply \u0026ldquo;Last Access\u0026rdquo;) is a user-level tool that shows which services were accessed by that user, when they were accessed, and which policy granted them that permission. This is great for maintaining least-privilege, because if a user has permissions to a service they never use, those permissions should be removed.\nBest practices # To close out the portion on Identity and Access Management, here are some tips and best bractices for securing your AWS account.\nDon\u0026rsquo;t use the root account. # Every AWS account comes with a pre-built super-admin account called the \u0026ldquo;root\u0026rdquo; account. This account can do absolutely anything, and cannot be restricted. It\u0026rsquo;s tempting to just use it, especially if you are the only one using the AWS account, but the cost is too high. If your root credentials leak, you might as well close your account because anyone with those credentials can do anything they choose. Instead, when you first create your AWS account, use the root account to create another IAM user and give them admin rights. Then enable MFA for the root account, make the password long and complex (like 25+ digits), and just leave the account there until you actually need it. Use the second user to actually manage your AWS account as an admin. If these credentials leak, that user can be removed, and a new admin can be created to take its place.\nUse the security tools AWS gives you. # AWS has many MFA options for your user acounts, use them. Set up roles and policies to enable least-privilege. Use Accesss Keys, especially for programmatic user account used in scripts and programs. Create one user per person, or per programatic access, don\u0026rsquo;t share credentials. AWS gives you all these tools for free with no limits so use them. Don\u0026rsquo;t be complacent, because that\u0026rsquo;s where mistakes are made and real problems arise.\nApply policies to groups, not users. # If a user needs a permission, they should be added to a group that has that permission. This makes permission management much more scalable and auditable. If they need a specific set of permissions that doesn\u0026rsquo;t exist in a group, create a new group. There is no limit to the number of groups or policies that you can create, so use them!\nAudit IAM frequently. # Creating a bunch of groups and policies can sometime sleave you with quote a messy set of overlapping permissions. Be sure to audit your users, groups, roles, and permissions frequently to ensure that stale accounts have been properly delt with, and permissions are appropirate accross your account. \u0026ldquo;Frequently\u0026rdquo; is a somewhat relative term that depends on the scale and turnover rate of your account. If you have two or three user accounts that don\u0026rsquo;t change permissions much, you could probably do an audit every 6 or 12 months and be safe. Not much is changing in that environment. But if you are creating and removing multiple accounts a month, monthly or even weekly audits may be more appropriate, depending on your risk tolerance. Your IAM should be as lean and efficient as you can make it!\n","date":"1 March 2026","externalUrl":null,"permalink":"/homelab/aws-saa-02/","section":"Homelabs","summary":"","title":"AWS Solutions Architect Associate - Pt2 (IAM)","type":"homelab"},{"content":"","date":"1 March 2026","externalUrl":null,"permalink":"/categories/certifications/","section":"Categories","summary":"","title":"Certifications","type":"categories"},{"content":"","date":"1 March 2026","externalUrl":null,"permalink":"/tags/journal/","section":"Tags","summary":"","title":"Journal","type":"tags"},{"content":"I have a modest homelab, only a few hundred gigabytes of content that doesn\u0026rsquo;t change often. Most of the content is not really that important, but I back it up anyways because, hey, storage costs are pretty low. I use BackBlaze B2, which is about $6/TB, which I am easily under. So you\u0026rsquo;ll image my surprise when my February bill came from BackBlaze, and it was for $103.91.\nYup, I went from spending about $2.50, to over one hundred dollars in about a month. How?\nMy first though was storage increase, because that was the main metric I think about. At the time I didn\u0026rsquo;t really put together that that would mean I would have to be storing almost\u0026hellip; 17TB of data, and I don\u0026rsquo;t even own enough drives to store that much at home, much less in the cloud. Nevertheless, this was my first, panicky, thought, so I logged into the management console and started investigating. I dug through the buckets and directories, looking for high numbers. I found that my uptime-kuma database was using about 900GB, which is ridiculously high. Drilling down I discovered that, actually, that 900GB was aggregated accross all the file versions of that database.\nWhen I set up BackBlaze I had enabled file versioning and set no lifecycle policy, meaning every version was kept forever. I had also, somewhat recently, started backing up everything every hour, because that sounded like a good idea. Additionally, my understanding of BackBlaze\u0026rsquo;s versioning scheme was incorrect, I thought that they only stored the difference between the versions, but instead they actually store an entire seperate copy, if the file has changed.\nUptime Kuma\u0026rsquo;s database includes all the metrics for each node it tracks, so it\u0026rsquo;s database does change frequently. It was currently sitting at just under 1GB on my server. So, every hour for the past month, I had been backing up this ~1GB file. Everytime the backup ran, the file had changed, so BackBlaze made an entirely new 1GB file version. You can see how this could esclate. Checking around there were a bunch of other log files that were unnecessarily large. I immediately turned off versioning for all my buckets. I never really needed it, it was just a nice to have, but not if it was going to cost me $100!\nI thought I had solved the problem but I decided to check my billing history, just to see when this price increase started. 2 months ago I saw my normal ~400GB storage cost, costing me about $2.50. Then last month it was increased to ~$14 and about 600GB. Then this past month, the full ~1TB, costing me $103. The storage increase was beause of the hourly backups and versioning, that part now made sense. But the final cost didn\u0026rsquo;t align, and it was at this this point that I remembered that BackBlaze only charged $6/TB, meaning I was being charged about $97 for something other than storage. I clicked on the most recent bill and looked at the details. Class C API transactions cost me about $97. Expanding that option I saw that the API call b2_list_file_names accounted for all of that and that it has been called over 1 million times.\n$97 for over 1 million API calls. I looked up what the API call does, and as the name implies, it lists out the files in a bucket. I assume this is part of the process when TrueNAS backs up the files, it first needs a list of the existing file structure, but the default way that it is done is to issue one API call for each file and folder. However, all the advice I found online recomended enabling --fast-list in the settings for each Cloud Backup job on TrueNAS. This reduces the number of API calls, while increasing CPU usage slightly because it recieves multiple file paths (up to 1,000) in one API call.\nSo of course I immediatly enabled --fast-list on all my backup jobs. I then looked through BackBlaze\u0026rsquo;s settings looking for alerts. I had wildly over spent, and was never alerted, why was that?\nBecause I haden\u0026rsquo;t enabled them! I had enabled storage alerts (\u0026ldquo;Send me an email if I spend more than $5 on storage\u0026rdquo;), but for Class A, B, and C transactions I had selected \u0026ldquo;No Cap\u0026rdquo;.\nSo of course I immediately enabled $5 caps for all of them, just to get started.\nTo be absolutely clear, none of this is BackBlaze\u0026rsquo;s, or even iXsystems\u0026rsquo;s (makers of TrueNAS), fault. I am a happy customer of each and will continue to use both. This was a case of an over-confident enthusiast who started messing with things he didn\u0026rsquo;t fully understand at the time.\nSo take this as a warning (I sure am), check your cloud costs, especially when you make major changes like I did. Turn on alerting for all categories, even ones you don\u0026rsquo;t think you\u0026rsquo;ll use.\n","date":"23 February 2026","externalUrl":null,"permalink":"/homelab/100-dollar-mistake/","section":"Homelabs","summary":"","title":"How I spent nearly $100 on useless API calls","type":"homelab"},{"content":"I recently became a AWS Certified Cloud Practitioner! The exam was relatively simple, it primarily involved understanding what all of AWS\u0026rsquo;s services do at a very high level. However, it doesn\u0026rsquo;t really prepare you to use the services, so I am continuing down the Solutions Architect Path with the Solutions Architect Associate certification!\nI found a highly rated course on Udemy called \u0026ldquo;Ultimate AWS Certified Solutions Architect Associate 2026\u0026rdquo; produced by Stéphane Maarek. In an effort to show my competency, retain the information myself, and perhaps even teach you a thing or two, I\u0026rsquo;ll be producing some journal-style blog posts showcasing what I have learned.\nSide note: anyone in the U.S. Air Force gets access to a bunch (all?) of Udemy courses for free through Digital University.\nIAM # In our modern world, an \u0026ldquo;account\u0026rdquo; is typically tied to an \u0026ldquo;identity\u0026rdquo;. So your Facebook or Google account are tied to your Facebook and Google identies respectively. No other account can access your identity. In the world of cloud computing, this coupling falls apart, for an AWS \u0026ldquo;account\u0026rdquo; is not an identity, but moreso a bin that can house various resources. This bin can then be accessed by various user identities, which themselves are resources housed within the AWS account bin.\nThe service which manages these user identities is IAM, the first section of Stéphane\u0026rsquo;s course.\nIAM or Identity and Access Management, is the AWS service that manages Users, Groups, permissions, and all the access policies used to access, add, remove, or modify the resorces in your AWS account.\nAn IAM User is an idividual identity used to access account resorces. This user can be a human user, or a progomatic user, such as one used in a script or program. For example, in my homelab I currently have a progromatic user that backs up my photos to an S3 bucket. The S3 bucket is a resorce, as are an EC2 virtual machine, a Lambda function, or Route 53 DNS entry (we\u0026rsquo;ll cover all these later on, stay tuned).\nAn IAM user can be part of any number of groups, includng zero. A group can only contain users, not other groups.\nPolicies # Both groups and users can have policies assigned to them. A policy is an Allow/Deny statement regarding a users level of access to a resource. A user or group can have any number of policies associated to it. In general, the most restrictive policy wins, though I haven\u0026rsquo;t tested this or researched too deeply to know exact policy execution order. Policies are represented as JSON objects and look like this:\n{ \u0026#34;Version\u0026#34;: \u0026#34;2012-10-17\u0026#34;, \u0026#34;Statement\u0026#34;: [ { \u0026#34;Sid\u0026#34;: \u0026#34;AdminAllowAll\u0026#34;, \u0026#34;Effect\u0026#34;: \u0026#34;Allow\u0026#34;, \u0026#34;Action\u0026#34;: \u0026#34;*\u0026#34;, \u0026#34;Resource\u0026#34;: \u0026#34;*\u0026#34; } ] } The Version field will always be 2012-10-17. I couldn\u0026rsquo;t find references to any other version except 2008-10-17, so presumably those represent dates, but I am not sure.\nThe Statement section is then a list of objects, each of which Allow or Deny certain access to certain resources. This example is very simple, you could get much more complecated.\nEach object includes a sid (I assume this stands for Security Identifier) which is an optional name for this object, it can be any name you want. Here, I have AdminAlowAll as an easily understandable name.\nUnder the sid is the Effect, which can be either Allow or Deny. This obviously controls whether the actions listed will be allowed or denied.\nThe Action section is a list of API actions that are being referenced. You can see here that I use a star (or asterisk) to represent all actions. Everything from access-analyzer:ListAnalyzers to xray:UntagResource and all the others in between (s3:ListAllBuckets, ec2:CopySnapshot and iotevents:DescribeAlarmModel to name but a few). These actions are all of the things that a user can do when interacting with a resource.\nLastly is the Resource section which specifies which resource these actions should apply to. Again, I have all resources selected here, by using the star. If you wanted to specifiy a specific resource, you would include its ARN or Amazon Resource Name. Some examples of ARNs are:\nAn S3 bucket: arn:aws:s3:::backup-photos A IAM Policy: arn:aws:iam::102381579309:policy/test-policy-name A VPC Subnet: arn:aws:ec2:us-east-2:122312579368:subnet/subnet-1f7cf9f1d81704b00 An SES Identity arn:aws:ses:us-east-2:881271468296:identity/mckay.one Virtually everything you create on AWS will have an ARN associated with it, and each service has many, many granular API actions so these policy definitions can grow to be long and convveluted. Best practice of course would be to break them up into smaller modular chunks, but sometimes that is not an option.\nI know I barely touched on actions and ARNs in this post, if you are interested in learning more, I suggest you do some more research, perhaps check out Amazon\u0026rsquo;s documentation!\nARNs: https://docs.aws.amazon.com/IAM/latest/UserGuide/reference-arns.html Actions: https://docs.aws.amazon.com/service-authorization/latest/reference/reference_policies_actions-resources-contextkeys.html Authentication Methods # Moving away from policies, actually authenicating as a user (whether human or otherwise) has a few different options:\nThe web console has a username and password, with optional (though recomended) MFA. The password requiremnets are customizable at the AWS account level from 6 digits and no special character requirments all the way up to 128 digits minimum and full upper/lower/number/special character requirements. AWS has a CLI client that can be installed on Windows, macOS or any distro of Linux. This CLI requires an Access Key and Secret Access Key, which are basically a username and password, but randomly generated by AWS. These keys can be created in the web console, and are unique for each account, so don\u0026rsquo;t go sharing your access keys with your colleages! Lastly, AWS has an SDK for JavaScript, Python, PHP, Node.js, Java, .NET, Go, Ruby, Arduino, C++, Embeded C, Android, and iOS. An SDK is a library of code that can be used while building a script or application. This also uses the same type of access keys as the CLI. Fun Fact: The AWS CLI is written purely in Python, and so uses the Python SDK to interact with your AWS Account.\nThere is a bunch more to cover in IAM, but that is all for this post. I hope you enjoyed it, and learned something useful!\n","date":"18 February 2026","externalUrl":null,"permalink":"/homelab/aws-saa-01/","section":"Homelabs","summary":"","title":"AWS Solutions Architect Associate - Pt1 (IAM)","type":"homelab"},{"content":"This is my first post regarding AWS EC2. This post is an introduction to what EC2 is and then some basic set up steps to get it up and running.\nEC2 Introduction # Virtual Computing is one of the cornerstones of modern computing. Virtualization allows for secure and efficient use of the massive resources that modern servers can provide. AWS offers a variety of virtualized computing services, but the first and most basic is Elastic Cloud Compute (EC2).\nEC2 was one of the first services that AWS launched with back in 2006/07. It allows you to run a virtual server in the cloud at the click of a button. Even though I understand how it all works behind the scenes, it\u0026rsquo;s still magical to be able to click-click-click and have a virtual server boot up in a data center somewhere.\nThe EC2 service is actually a collection of distinct, yet related, services. They are:\nEC2 Instances. These are the actual virtual machines. EBS (Elastic Block Storage). These are the drives that get attached to an EC2 Instance to store data. ELB (Elastic Load Balancer). This is a service to distribute incoming network traffic between multiple VMs. More on this in a later post ASG (Auto-Scaling Group). This is a service that, once configured, will automatically create or destroy new VMs as neccesssary. More on this also in a later post. Setting Up an EC2 Instance # Every computer needs 4 things; a CPU, memory, storage, and a set of instructions (in this case, an operating system). An EC2 instance is no different. When you begin to launch the instance, one of the first questions you\u0026rsquo;ll be asked is which Amazon Machine Image (AMI) you want to launch with.\nIn its most basic state, an AMI is just an opperating system. But unlike an OS, an AMI can include prconfigured software. Amazon has a whole bunch of options, from basic OSs like Linux, Windows, and even macOS, to network appliances like firewalls or Kubernetes nodes. They also have an image builder, so if you need a fully custom AMI, you can build that yourself.\nOnce you have the AMI selected, it\u0026rsquo;s time to select the Instance type, which combines the CPU and memory. AWS has a number of instance familes, each of which is designed for specific types of tasks\nGeneral Purpose: These offer a balance of CPU and memory. As the name implies, these are the go-tos for the average computing needs. Compute Optimized: These offer more CPU cores with less memory, as compared to the general purpose options. These instances excell at high volume work such as batch procesing, or media transcoding. Storage Optimized: These offer millions of low-latency, random I/O operations per second, perfect for time-sensitive opperations such as databases, or certain data processing applications. Memory Optimized: these offer more memory than the general purpose options, with fewer CPU cores. These are great for in-memory databases, or certain analytics programs. HPC Instance: These instances are best suted for high performace tasks such as complex model simulations or deep learning Accelerated Computing: These instances have hardware accelerators, or co-processors, to perform functions such as precise floating-port opperations or graphics processing more efficiently. Each one of these familes have many pre-configured options. You can browse through them at https://instances.vantage.sh/\nLastly, after youve selected the AMI (operating system) and the instance type (CPU \u0026amp; RAM), your VM will need durable storage. That\u0026rsquo;s where EBS comes in. It provisions and manages all the virtual drives in your account. Obviously your EC2 Instance needs a main/root drive for the opperating system, but you can also add more drives, just like you can in a physical server.\nEBS Volumes come with a few different options, which all boil down to \u0026ldquo;how fast and how performant do you want your drive\u0026rdquo;. The options span the gamut from infrequent cold storage (HDD-based st1) to general purpose SSDs (gp3) to maximum performance for critical databases (io2 Block Express).\nAfter selecting the AMI, Instance type and creating an EBS volume, you technically have everything you need. However, the VM will be isolated because we havent said anythig about networking yet. We\u0026rsquo;ll talk later about VPCs and network subnets, for now those can just be set to defaults. However, one thing we do need to talk about is security groups!\nSecurity Groups are collections of firewall rules. Their main advantage is the fact that they are a seperate entity from the EC2 instance, so you can apply the same set of firewall rules to multiple EC2 instances. Just like any firewall ruleset you can specifiy incoming or outgoing ports and IP addresses. When a security group is first created, it has a default rule to allow port 22 (SSH) from any IP address, and to allow all outbound ports/IPs from the VM. You can (and obviously, should) change this to suit your needs. This is the default just to get you up and running via SSH.\nAfter this your EC2 instance is good to go! As with anything, there are several additional options for advanced usage such as joining a domain, changing shutdown behavior, or any other specific requirments you may have. An option that I only recently discovered was the \u0026ldquo;User Data\u0026rdquo; section. Here you can add a script to run durring the first boot of machine. This can be used to update the package repository and install specific programs, or anything else your workflow requires on first boot. Note: this script ONLY runs on first boot, every boot afterwards will be normal and will not execute this script.\nWell that\u0026rsquo;s enough about EC2 for now, but that\u0026rsquo;s not all there is to say! Stay tuned for more on this topic!\n","date":"18 February 2026","externalUrl":null,"permalink":"/homelab/aws-saa-03/","section":"Homelabs","summary":"","title":"AWS Solutions Architect Associate - Pt3 (EC2)","type":"homelab"},{"content":"Welp, I\u0026rsquo;m ready to start writing my backup script!\nI ended up using Python, mostly because I feel more comfortable using it. I considered Bash, and perhaps in the future I will convert the process to Bash. But as I said, I am more comfortable in Python, and, at least for now, think Python could be better. We shall see. Bash may be more ubiquitous, but I\u0026rsquo;m not planning on ever running this on any exotic hardware, so Python is a reasonable requirment.\nAt first, I had one main monolithic script that interacted with every application. This was \u0026ldquo;easier\u0026rdquo; because it was a single file to edit, but it got very busy very fast. It meant that if I ever changed the containers that I had, I would have to scrub through and remove all references to that container. Additionally, I eventually want to do rotational backups (daily, weekly, monthly, etc), so thinking about coding all that logic into this main script became a bit overwhelming. I ended up breaking it up so one backup script handles one applications requirements. Each of these individual backup scripts will be called one by one from a main script. Currently that main script is just a Bash script that calls each Python file like so:\n#!/bin/bash /docker/authentik/backup.py /docker/calibre-web/backup.py /docker/collabora-code/backup.py . . . This allows for flexability with each application\u0026rsquo;s backup needs. It means that each applications\u0026rsquo;s backup needs are self-contained and not intertwined with any other application, so if I remove a container, all I have to do is remove the call in the main script, and it won\u0026rsquo;t get backed up. Hopefully, this will scale well and be more maintainable. Only time will tell.\nWithin each of these backup scripts I have logic for backing up the database (if applicable), deleting any databse backups older than 30 days, and encrypting the .env files. Since these tasks are standard accross the containers and the logic is the same, I created a Python psudo-module that contains all this logic that is simply imported in each script. I couldn\u0026rsquo;t find out exactly how to make it an actual module and perform a clean import, so I end up modifying the Python PATH first with this code:\n#/docker/authentik/backup.py #!/usr/bin/python3 import sys sys.path.append(\u0026#39;/docker/backup\u0026#39;) import backup_utils as bu Then in /docker/backup I have a file called backup_utils.py that contains all the common functions. I will eventually push all my Docker config files to my GitHub, so you can view them there.\nIt was also at this time that I started using git. I had used it before, mostly through VSCode, never really through the command line. I won\u0026rsquo;t go into detail here since all I did was basics like git init, git status, git branch, git checkout {branch}, git add {file}, and of course git commit -m {messsage}.\nWhen I innitially set about doing this project I thought I could create TrueNAS snapshots via API. While that is true (sorta, they use websockets which is a whole different thing than I was expecting), these manually created snapshots are not part of any automated lifecycle tasks and so would have to be manually deleted (through either the GUI or API). I decided that was too much complexity, so instead I will simply schedule the script to run before the snapshots, that way the databases are as consistant with the filesystem as possible.\nWith all that said, stage 2 of my backup plan is complete! Next on the list is actually storing all these backups well.\n","date":"15 February 2026","externalUrl":null,"permalink":"/homelab/back-it-up-2/","section":"Homelabs","summary":"","title":"Back It Up - Part 2","type":"homelab"},{"content":"Backups are like oral hygiene. Everyone knows they\u0026rsquo;re important and most people are doing something, but very few people are actually putting in the effort to do it completely.\nThey\u0026rsquo;ve always been on my TODO list, but other things have always caught my attention and distracted me from this all-important endeavor. Plus, I was honestly kinda scared to dive into the proper way of doing Docker Volume and database backups; everything is working fine for now, why change it?\nI recently had a scare in my homelab that made me re-think how I have things architected and how effective my backups actually are. This is a tale as old as time; the problem doesn\u0026rsquo;t get fixed until it NEEDS to get fixed. Luckily for me, no data was lost, and I\u0026rsquo;m on track for some major improvements!\nThe Incident # I run a Nextcloud instance, and due to some poor configuration on my part I thought I had corrupted some data. I was playing around with some MINIO S3 buckets (a story for another time) and I mapped one to my Nextcloud account as an external device. I wasn\u0026rsquo;t using it for anything, I just saw it as an option and wanted to play around. Then, later, I was cleaning up said buckets and must have deleted the access keys that I used to map it to Nextcloud. Instead of simply giving an error stating \u0026ldquo;failed to connect to S3 bucket\u0026rdquo;, Nextcloud apparently completely errors out and displays an empty account. I was able to log in no problem, but none of my files were displayed (the screen just said \u0026ldquo;no files in here\u0026rdquo;), none of my calendars or contacts loaded; I seriously thought I had (somehow) bricked my installation! I was beside myself because I had no idea what I could have done. No major changes have happened, no updates, no re-configuration (besides the S3 bucket, which I had forgotten about), nothing. In a panic, I checked the folder I have mounted in the container that actually stores all the data and\u0026hellip; all the files were there. That\u0026rsquo;s strange, I wonder why Nextcloud isn\u0026rsquo;t recognizing them? As I said, databases scared me, so I had never gotten around to actually backing it up, but I did have a backup of the entire virtual machine, that should work!\nIt did not.\nI then tried to roll back the dataset where I store the data files (using ZFS on TrueNAS). That also didn\u0026rsquo;t work. I contemplated downloading the files from BackBlaze, and just starting over from scratch, but then I thought to check the server logs. That\u0026rsquo;s where I saw some error about an S3 failure. Thankfully, I was still able to access that configuration, I deleted it, and everything came back up.\nWeird.\nBut it got me thinking, I didn\u0026rsquo;t have a solid process for restoring from a failure. I did have backups (I\u0026rsquo;m not a crazy person alright), but I didn\u0026rsquo;t have a solid process. I was backing up the data to an S3 bucket in BackBlaze, but I wasn\u0026rsquo;t backing up the database or the config files. I was backing up the VM, but none of the individual containers. And despite using Docker Compose (as a psudo IaC option), I wasn\u0026rsquo;t using git, so any changes were pretty much permanent.\nI decided enough was enough. Let\u0026rsquo;s do this the right way!\nThe Plan # So I did what any person would do nowadays, I turned to ChatGPT. It suggested some ideas, I added some others, asked some questions, did some research and finally, this is the plan I\u0026rsquo;ve come up with:\nAll databases will use Docker Volumes. A script will be created to generate backups of these databases daily. Since I prefer to use Postgres, this can be done while the databae is running with no interuption. All other volumes will be NFS mounts from TrueNAS mounted via Docker NFS mounts (not mounted to the host). Since TrueNAS uses ZFS, snapshots of these dataset\u0026rsquo;s can be taken, and the datasets can also be backed up to BackBlaze S2. These snapshots can be programmatically taken through the same script that backs up the databases. If a container\u0026rsquo;s volume consists of purely configuration files, those will be bind mounted and versioned with git. All environment variables will be placed in .env files and encrypted. All Compose, .env, and other config files will be versioned with git and backed up to GitHub. This should give me the granular ability that I want. I\u0026rsquo;ll be able to surgically recreate an individual container, instead of affecting all containers at once. Once this is fully implemented, I\u0026rsquo;m planning on spinning up a test VM and will attempt to actually recreate my production VM.\nThe Implementation. # Some containers were easy to get setup with this new system. Traefik, for example, (my reverse proxy), only uses configuration files, no real volume needed. It was already setup with its config files on a bind mounted folder on the host, so it was ready to go immediately. Others, however, are a bit more complicated. Nextcloud for example.\nI am using the LinuxServer.io image for Nextcloud. This is a wonderful image that has two volumes, data and config. The config volume I had previously setup as a regular Docker Volume, while the data volume was bind mounted to a folder on the host that itself was an NFS mounted dataset from TrueNAS. I wanted both of these to be NFS shares, and both to be NFS Docker Volumes, cutting out the middle man and connecting the container directly to the NFS server. This would also fix a minor problem I\u0026rsquo;ve had where when the host rebooted, the container would start before the NFS share was mounted, requiring me to manually restart any container that relied on NFS.\nTo make the config volume an NFS share, I would obviously need to export the data from the Docker Volume and move it to the NFS server. That was easy enough to do with the following commands:\ndocker exec -it nextcloud tar -cf config.tar /config docker copy nextcloud:/config.tar ./config.tar The first command create a tar file (the Linux version of a zip file) of the config directory. The second command then copies that file out from the container to the host. Both of these commands need to be run while the container is running. After I copied the config.tar file out, I stopped both the nextcloud server and database containers.\nI then created two child dataset on TrueNAS under my top-level Nextcloud dataset (the one currently holding just the data). These child datasets were config and data. I copied the tar file into the config dataset, un-tar\u0026rsquo;ed it, then copied the data into the data dataset.\nOf course, I had a weird issue with the NFS folders. Since NFS treats each dataset as it\u0026rsquo;s own file system, you can\u0026rsquo;t mount both the parent and child datasets via NFS. I had the datasets pool/Nextcloud, pool/Nextcloud/config and pool/Nextcloud/data all mounted with NFS on the one host. When I transfered the config file into the config dataset, TrueNAS registered the increased size in the parent dataset (pool/Nextcloud), not the child dataset (pool/Nextcloud/config). But navigating through TrueNAS\u0026rsquo;s shell, the config.tar file was nowhere to be found. It was really weird, and caused me even more stress (and lead to a few snapshot rollbacks). I then found this post where the poster had the same problem. Thankfully the solution was easy, only share the child datsets. I did this, mounted them to the host, transfered all the relevant data, then unmounted them from the host and edited my Docker Compose files to include two new volumes\nservices: app: image: lscr.io/linuxserver/nextcloud volumes: - nfs_config:/config - nfs_data:/data ... volumes: nfs_config: driver_opts: type: nfs o: addr=truenas.mckay.one,rw,nfsvers=4 device: \u0026#34;:/mnt/pool/nextcloud/config\u0026#34; nfs_data: driver_opts: type: nfs o: addr=truenas.mckay.one,rw,nfsvers=4 device: \u0026#34;:/mnt/pool/nextcloud/data\u0026#34; Alright, cool! Everything should be setup, just a quick docker compose up -d and we\u0026rsquo;ll be on our\u0026hellip;.\n502 Bad gateway\nChecking server logs, a whole bunch of \u0026ldquo;unable to write file, permision denied\u0026rdquo;\nWhat? Why?\nUsing docker exec, I checked the permissions in the container, and all the data and config files were correct.\nAt this point I was (proverbially) ripping my hair out. I tried a few other things, but nothing panned out. I was a bit desperate, and frantic at this point. I was so close, but I also felt way out of my leauge. So, once again, I turend to ChatGPT (one of these days I\u0026rsquo;ve gotta get a local LLM setup\u0026hellip;) ChatGPT suggested that TrueNAS was applying root_squash to the mounts. I remember reading about that a few years ago. Thankfully, ChatGPT explained it pretty well\nYour container runs as UID 1000, and your files on NFS are owned by 1000:1000, which is good.\nBUT you\u0026rsquo;re mounting the NFS share as root (UID 0) inside the container — because Docker mounts volumes as root — and TrueNAS is most likely applying root_squash, which maps root → nobody (UID 65534).\nSo even though the files look like they\u0026rsquo;re owned by 1000:1000 inside the container, all operations coming from root get mapped to nobody when they hit the NFS server.\nThis results in:\nReads may work Writes fail Ownership appears right inside the container, but NFS silently denies operations This is exactly what you’re experiencing.\nThat made sense! Since Linux relies heavely on UIDs and GIDs, the actual number of the user or group is important. UID 0 (the root acount) in the container was being mapped to user 65534 on the TrueNAs system, also known as nobody. This nobody user does not have access to the pool/nextcloud/config dataset on TrueNAS, so write opperations were failing. So all I had to do was explicently map the containers root user to TrueNAS\u0026rsquo;s root user This can be done via the \u0026ldquo;Maproot User\u0026rdquo; and \u0026ldquo;Maproot Group\u0026rdquo; settings in the Advanced NFS share settings in TrueNAS.\nNOW just a quick docker compose down \u0026amp;\u0026amp; docker compose up -d\u0026hellip;\nAnd it works! All of my files are there, as well as all of my contacts, calendars, and settings!\nConclusion # Well I learned a lot this go-around. I learned about NFS file permissions, NFS Volumes for Docker, tar-ing files and folders, Postgres\u0026rsquo; safe backup opperations, and the docker cp command. Much more is to come of course. I haven\u0026rsquo;t actually backed anything up yet, just started setting up the framework to do so. Stay tuned for more on this topic!\n","date":"10 December 2025","externalUrl":null,"permalink":"/homelab/back-it-up/","section":"Homelabs","summary":"","title":"Back It Up - Part 1","type":"homelab"},{"content":"I recently followed a tutorial put out by TechnoTim called \u0026ldquo;PostgreSQL Clustering the Hard Way\u0026rdquo;. It is a fantastic video, he does a good job of walking you step by step though the instructions all while explaining what each thing is doing and what each line of configuration means. After following his tutorial and having my own HA PostgreSQL cluster, I wondered if I could do the same with some webservers. Certainly it should be easy, right? After all, a simple webservers should be the easiest thing to configure because they\u0026rsquo;ve been around as long as the internet has.\nMy methods in this post are not production ready. This is a quick-and-dirty project I did as a proof of concept; a real HA webserver should have TLS and proper Apache2 sites configured, as well as probably some other things I don\u0026rsquo;t know about.\nPrerequisite Configuration # Some prerequisites before we get started. I have four servers setup as VMs on a Proxmox cluster. They are all fresh installs of Debian.\nServer IP application SVR-01 192.168.100.10 apache2 SVR-02 192.168.100.11 apache2 HA-01 192.168.100.20 HAProxy, keepalived HA-02 192.168.100.21 HAProxy, keepalived The way this will work is keepalived will create a virtual IP (VIP) address that our clients can connect to, in my setup that will be 192.168.100.60. The MAC address that this IP will resolve to (and therefore the machine that our clients should send their requests to) will change depending on who keepalived determines is the \u0026ldquo;MASTER\u0026rdquo;, at first this will be HA-01, but if it goes down, then HA-02 will become the master. haproxy on whichever node is the \u0026ldquo;MASTER\u0026rdquo; will then equally balance the load between SVR-01 and SVR-02.\nSetting up my own HA webserver # To have a highly available webserver, one needs multiple webservers. I reused the same VMs I had setup for TechnoTims tutorial (SVR-01 and 02) and just installed Apache2 alongside PostgreSQL.\nsudo apt install apache2 Then, just to make sure it installed correctly, I navigated to the respective IP addresses for SVR-01 and SVR-02 in my web browser. On both I saw the default Apache2 page, so I knew the installation had succeeded.\nOrdinarily, these two servers would have the same content (synced using rsync periodically perhaps), but I changed the default index.html page to include the server name so I could differentiate between the two. I put the following in /var/www/html/index.html, changing the svr-0x number appropriately.\n\u0026lt;html\u0026gt; \u0026lt;body\u0026gt; \u0026lt;!-- Change this number depending on the web server name --\u0026gt; \u0026lt;h1\u0026gt; This is svr-01 \u0026lt;/h1\u0026gt; \u0026lt;/body\u0026gt; \u0026lt;/html\u0026gt; I checked both once again, and they both showed my simple page.\nNow onto haproxy. It can be installed through the package manager as well.\nsudo apt install haproxy\nOnce it was installed we can add the configuration. I was able to figure out by just resuming most of TechnoTim\u0026rsquo;s configuration. I put the following in /etc/haproxy/haproxy.cfg on HA-01\nfrontend apache_frontend bind *:80 mode http default_backend apache_backend backend apache_backend mode http server apache-01 192.168.100.10:80 check server apache-02 192.168.100.11:80 check The frontend section creates the receiving part, this is the port and configuration that HAProxy is listening for. The backend section is then what HAProxy will do with that incoming request. In this case, simply forward it on to one of the defined servers. The check parameter at the end of each server definition tells haproxy to periodically health check the endpoint. More information in that is available on HAProxy\u0026rsquo;s Blog\nThen a quick restart for HAProxy\nsudo systemctl restart haproxy Then navigating to the IP address of HA-01 I saw the page for SVR-01. Refreshing the page quickly a bunch of times forced it to load balance and so I also saw the page for SVR-02! Everything was working! I then copied the same configuration to HA-02.\nThe last thing to get working is the Virtual IP (VIP) using keepalived, so that haproxy doesn\u0026rsquo;t become a single point of failure. This took a little bit more investigation. I am thankful for the information at this site.\nFirst, install keepalived on HA-01\nsudo apt install keepalived Then add the following to /etc/keepalived/keepalived.conf\nvrrp_instance web_server { state MASTER interface eth0 virtual_router_id 20 priority 100 advert_int 1 virtual_ipaddress { 192.168.100.60 } } Copy the same to HA-02, changing the state to BACKUP and the priority to any number lower than 100. Then restart keepalived on both HA-01 and HA-02.\nsudo systemctl restart keepalived Navigating to 192.168.100.60 and quickly refreshing the page leads to the same result as earlier. This means our VIP was correctly resolved to HA-01\u0026rsquo;s MAC address, which took our request and balanced it between SVR-01 and SVR-02. We can test this even further by shutting down the haproxy node that is the master, in this case HA-01. keepalived on HA-02 should notice the drop out and assign itself to be the new \u0026ldquo;MASTER\u0026rdquo; and haproxy on HA-02 will then proxy the request to the webservers. If you had more than two keepalived nodes, then the highest priority value always becomes the new \u0026ldquo;MASTER\u0026rdquo;.\nHA Wacky Webserver # While doing the above, I realized that keepalived has no interest in port numbers, it is Layer 3 only, and haproxy has definitions for port numbers for each server declaration. Would it be possible then to have two webservers on different ports (say, 9080 and 9081) get routed though a virtual IP that was itself listening for web traffic on a different port (say 9090)? Only one way to find out!\nFirst I needed to make some web service available on a separate port. I looked into the Apache2 documentation and saw they you can define multiple Listen interfaces, as well as multiple VirtualHosts.\nSo, on SVR-01 in /etc/apache2/ports.conf I added a second Listen directive. Now this VM is listening on both port 80 and port 9080.\nListen *:80 Listen *:9080 Then in /etc/apache2/sites-available/000-default.conf I added a second VirtualHost. This defines what should happen when a request comes in on a particular IP:Port combination. Here we are telling Apache that a request that comes in on any IP address defined on this host over port 9080 should receive the contents in /var/www/html2\n\u0026lt;VirtualHost *:9080\u0026gt; ServerAdmin webmaster@localhost DocumentRoot /var/www/html2 ErrorLog ${APACHE_LOG_DIR}/error2.log CustomLog ${APACHE_LOG_DIR}/access2.log combined \u0026lt;/VirtualHost\u0026gt; Then, to create the new html2 directory and index file\nsudo mkdir /var/www/html2 sudoedit /var/www/html2/index.html \u0026lt;html\u0026gt; \u0026lt;body\u0026gt; \u0026lt;!-- Change these numbers depending on the web server --\u0026gt; \u0026lt;h1\u0026gt; This is wacky-svr-01 port 9080 \u0026lt;/h1\u0026gt; \u0026lt;/body\u0026gt; \u0026lt;/html\u0026gt; I did similar things on SVR-02, except everywhere I put 9080 for the port number on SVR-01, I put 9081 on SVR-02. I was then able to access each new page at their respective IP:Port combos.\nThen, on HA-01, I just had to add one new frontend and backend\nfrontend wacky_frontend bind *:9090 mode http default_backend wacky_backend backend wacky_backend mode http server wacky-01 192.168.100.10:9080 check server wacky-02 192.168.100.11:9081 check Notice how I defined the wacky_frontend to be listening on port 9090.\nAccessing HA-01\u0026rsquo;s IP address and port 9090 in my web browser brought be to the webpage for wacky-01. Once more, spamming the refresh button also showed the wacky-02 page. I was able to load balance between two different machines using three different ports! Pretty cool!\nOf course this could also be extended to include HA-02 and a new keepalived IP address. I decided not to do that.\nConclusion # I have known about load balancing and Virtual IPs for a while now and always kind of assumed it was difficult or complicated to setup. I\u0026rsquo;m glad to see that for simple projects like this, it really wasn\u0026rsquo;t that difficult. This gives me the courage to add Virtual IPs to more services in my homelab, such as DNS.\nChallenge: Can you configure haproxy to not load balance equally? Maybe change it from the default of (I believe) 50/50 to something like 90/10. Or change it to simply failover when one webserver goes down.\n","date":"20 December 2024","externalUrl":null,"permalink":"/homelab/high-availability/","section":"Homelabs","summary":"","title":"Dipping my toes into High Availability","type":"homelab"},{"content":"","date":"3 December 2024","externalUrl":null,"permalink":"/tags/hardware/","section":"Tags","summary":"","title":"Hardware","type":"tags"},{"content":"","date":"3 December 2024","externalUrl":null,"permalink":"/tags/migration/","section":"Tags","summary":"","title":"Migration","type":"tags"},{"content":"","date":"3 December 2024","externalUrl":null,"permalink":"/tags/torubleshooting/","section":"Tags","summary":"","title":"Torubleshooting","type":"tags"},{"content":"","date":"3 December 2024","externalUrl":null,"permalink":"/tags/truenas/","section":"Tags","summary":"","title":"Truenas","type":"tags"},{"content":"This post was originally posted on a Wordpress blog that I had and forgot about. It is being posted here for completeness of my homelab journey. It was originally posted in the summer of 2024.\nThis is the first in a series of articles detailing my 2024 homelab update.\nI have been running my current homelab off an old 2013 or 2015 iMac I got from my grandfather, as well as a repurposed gaming PC I built in 2020. The iMac has been my application server (using Docker and Docker Compose to manage the applications), and the gaming PC has been my storage server, running TrueNAS CORE, which you may know is FreeBSD based. Nothing wrong with FreeBSD per se, but I am rather partial to Debian, so when I saw that TrueNAS SCALE was Debian-based, I knew I wanted to switch. Additionally, the 8GB of the iMac was starting to be a limiting factor, and I also wanted to do things “more correctly” like using a hypervisor, Kubernetes, and more Infrastructure-as-Code (IaC).\nI decided that the first thing to do was to upgrade my storage server, because once I got that taken care of, I could use it as a storage target. The main problem with my current computer was that it only had space for two HDDs, and I had three, so one was just propped up inside the case. It is a NAS-specific drive, but still, I wanted to give it a better life.\nInitial Testing and Officelabing # The first step in any good upgrade path is choosing the new system and validating that it works for your given needs. My needs were simple:\nSupports Debian Have at least 3 spaces for hard drives. Since Debian is over twenty years old, my first point was pretty much taken care of. As for the second, my day job is as desktop support for two departments at a large university, so there is a pretty steady flow of used computers that we get rid of. After getting the approval of my manager, I choose a Dell Precision T1700. It had spaces for four drives, and had support for 32Gb of non-ECC RAM.\nI took a few old hard drives we had lying around at the office, loaded up TrueNAS SCALE, and played around with it in my spare time for a few months. I practiced exporting and importing pools, poked around the UI, and generally got used to the, in my opinion, better interface that SCALE offers over CORE.\nThen school let out and the summer slump of my university job hit. Add to that, my wife was going to be out of town for a week, and I had plenty of time to bring this new computer home and initialize my new storage server. Things had been going just fine in my test environment, so I was excited to finally get that third hard drive properly mounted and protected.\nDEGRADED # Ever since I set up my homelab, I had been making backups (some to BackBlaze B2, some to AWS Glacier). Given that, I wasn’t too worried about my data being lost. Worst case scenario, everything breaks and my data is available in the cloud, but not locally accessible.\nSo I brought this new-to-me computer home, exported my current ZFS pool and shut down my gaming-turned-storage computer. I connected my three hard drives and an SSD for the boot drive to the Precision and booted into TrueNAS SCALE. It installed wonderfully, and I was met with the new dashboard I had come to like. I imported my pool, which seemed to go well until something caught my eye…\nPool Status: DEGRADED ⚠️\n“Degraded? How could that be?” I hadn’t had any errors (ever) with TrueNAS before, so I wasn’t sure the severity of the word “degraded”. I clicked on my pool information and saw that one of my three drives had not been recognized 1. Thankfully, TrueNAS uses ZFS and I had a RAIDZ1 VDEV, so no data was lost. But the health was obviously “degraded”.\nPerplexed by this, I exported the pool again and loaded my three drives back to my old server with CORE.\nPool Status: Healthy ✅\nMy old installation was able to read the drive, so it wasn’t a problem with the drive? I ran a short S.M.A.R.T. test on all the drives just to be sure, and it came back with no errors found. Weird.\nThus precipitated a two-day trek through forum posts. What was supposed to be a quick drive switch, turned into a troubleshooting nightmare where nothing would work. One of the first things I found was the 3.3v issue people were having with shucked drives. Mine weren’t shucked, but I gave it a shot. Nothing. I made sure the Precision’s BIOS was up-to-date, it was. The “faulty” drive was discoverable to both my old CORE installation, and to my MacBook (via a USB dock), but not to this Dell Precision. I checked the BIOS of the Dell and intermittently saw only two drives or all three drives show up (usually just the two). I was besides myself.\nAfter two days of troubleshooting, I gave up with that computer and brought home a second. This was a Dell OptiPlex 9020. I could only get 16GB of non-ECC RAM, but it had the same 4 HDD bays as the Precision.\nPool Status: DEGRADED ⚠️\nDegraded again. On a different system entirely with, presumably, a different enough BIOS. The OptiPlex BIOS also failed to register all three drives.\n“This is why people have imposter syndrome.” I thought to myself. “Because sometimes they really don’t know what is going on…”\nAt this point, I had accepted that I was going to have to use the same hardware, but I at least wanted the HDD bays of the Dell pre-built. I carefully dismantled first the pre-built, and then my old gaming/storage computer and transferred the motherboard. Alas, while my standard ATX motherboard fit, the front-panel connectors are some Dell-proprietary connector. I could probably make it work, but I was so discouraged at this point that I just rebuilt my old setup, this time with TrueNAS SCALE. My third HDD is now is a slightly better position, laid flat on the power supply.\nLessons Learned # Another good exercise after any kind of deployment or upgrade is to look back and see what could have gone better and what information can we carry into the future.\nThe good # While I didn’t exactly plan for downtime, the timing worked out that I would be able to take down my servers for an extended period of time, and would be able to devote sufficient time to fixing any problems. Additionally, I did leave my initial TrueNAS CORE installation intact, so that I could revert to it in the case of a failure (which I ended up having to do). Lastly, my troubleshooting skills came in handy when I had to nail down exactly what the problem was.\nThe bad # I didn’t mention this earlier, but one thing I failed to do was to make a backup immediately before I started the swap. My backup schedule was once weekly, and I started this project on a Wednesday, so I could potentially have lost ~4 days of data. In my scenario (where I am the main user and not much data changes) this was acceptable, but part of the purpose of my homelab is learning industry best practice, and “backups before migration” is definitely a best practice.\nLooking forward # I don’t know the next time that I will do a migration like this. It’s very probable that the hardware I have will stay the same for a while, with the possible exception of a new case to hold more drives (maybe an all SSD pool!). Regardless, though, next time I do anything major with my TrueNAS Server, I’ll be sure to take a backup of at least all my critical files, even if I can’t wait to play sysadmin again.\nThe drive in question was a TOSHIBA N300 4TB NAS drive, model HDWG440. The two drives that were recognized were TOSHIBA N300 4TB NAS drives, model HDWQ140. Details, details…\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","date":"3 December 2024","externalUrl":null,"permalink":"/homelab/truenas-migration/","section":"Homelabs","summary":"","title":"TrueNAS Migration","type":"homelab"},{"content":"This post was originally posted on a Wordpress blog that I had and forgot about. It is being posted here for completeness of my homelab journey. It was originally posted in early 2023.\nI’ve decided to layout what my homelab looks like now because this summer I hope to change some things.\nStorage # My current storage server is one that started life as a gaming computer that I built back in 2020. I had gotten a wild itch to build a computer (my first ever build), that admittedly I didn’t really plan much out. I also didn’t really factor in the fact that I am not a gamer. So after I built it I toyed around with Cities:Skylines, Stardew Valley, and other such games, but the computer largely sat unused (or at least under-used).\nThen I discovered homelabing, and suddenly I had a need for networked storage. I purchased three 4TB Toshiba drives and loaded up TrueNAS Core and had myself a fancy little Network Attached Storage server. The computer case I had was really only meant for one hard drive, so two of my hard drives are currently resting in some fairly precarious positions inside the case.\nApplications # What is homelabing without applications? I am currently running Ubuntu Server on an old iMac my grandfather gave me. On that server, I am using Docker as the container engine to run the various applications. Along with that, I am using Docker Compose as my “Infrastructure as Code”. Most of my docker containers use Docker Volumes to store data, but for those that store a “significant” amount (which is a metric I do not have a definition for) I use bind-mounts that connect back to my aforementioned TrueNAS server via NFS.\nWhen I first started on this homelab journey, I went a little wild, deploying every application I came across, whether I actually had a need for it or not. After a while (and my wife’s need for a more present husband), I stopped deploying every container under the sun and settled into actually using those I had a use for, and getting rid of those I don’t. Currently, the ones I use are:\nNextcloud w/ Colabora Office: Google Drive replacement Immich: photo storage Authentik: authentication/SSO Firefly-III: personal finance management Homepage: links and mini dashboard Mealie: recipe management Paperless-NGX: virtual filing cabinet Traefik: load balancer/reverse proxy code-server: VS Code environment to access all my docker compose files Dozzel: live log viewer FreshRSS: RSS feeder. I use NetNewsWire on iPhone to actually read articles Homebox: physical inventory management Network # I recently upgraded my network gear from the combo-box that Spectrum gave me to a TP-Link OMADA switch and access point with an OPNSense router. I do have three separate VLANS (Trusted, IoT and Guest), but I haven’t actually done any real configuration, so they’re not providing any security benefits.\nFurthermore, I am also using Cloudflare DNS as my only DNS service. This means, if I want a DNS record, I must expose the application to the internet. When I was still using Spectrum’s combo box, I tried setting up PiHole as a container, but for some reason I wasn’t able to get it to work. I haven’t exposed every service to the internet, so some services are still referred to by IP:Port and if I didn’t have Homepage, I would definitely forget which ones were which.\nSummer plans # My goals for this summer (2024) are modest, but quite substantial, given the current state of things:\nProper network setup\nFirewall Rules to restrict IoT movement/connections and fallout Internal DNS server so that every service can have a name and I can stop referencing to IP:Port. Route everything with Tailscale (one of the coolest gosh-darn products to ever exist) Transfer to “new” servers. As previously stated, I’m currently running everything off an old iMac. This works, But I’m hitting the limit of what’s possible with its 8GB of RAM. I’ll transition to a more “production” environment by using Proxmox and running everything in Virtual Machines. This will give me the ability to more easily create snapshots/backups and spin up/down resources as needed.\nTransition to use more IaC. Infrastructure as Code is such a neat concept. The idea that my entire stack, from hardware to application, can be described and deployed via a configuration file, which can then be saved, stored, tracked via change control and all the rest, is such a wonderful idea and I want to be part of it. This will probably start off with just using a bit of Ansible and vanilla Kubernetes, but my eventual goal/dream is to learn NixOS and Terraform/OpenTofu and be able to recreate everything with as minimal intervention as possible. Additionally, this would make it easier to create an “emergency script” so that in the (hopefully very distant) future when I die and my family takes over my homelab and network, they would be able to revert it to a “simple” state, or even clean everything up to dispose of/sell.\nWell, that’s about all I have. I’ll have more posts as I preform all the changes I just mentioned. Hopefully I’m able to accomplish all of them.\n#SoliDeoGloria\n","date":"2 December 2024","externalUrl":null,"permalink":"/homelab/2023-homelab/","section":"Homelabs","summary":"","title":"2023 Homelab","type":"homelab"},{"content":"","date":"2 December 2024","externalUrl":null,"permalink":"/tags/tour/","section":"Tags","summary":"","title":"Tour","type":"tags"},{"content":"Welcome to my site! Here I will be talking about the things that interest me.\nThe purposes of this site are three-fold:\nInformation archive for myself. I like to homelab and selfhost, and sometimes I forget the things that I have done, or how I did them. Ths site will be a reposatory of all my adventures Information archive for others. The internet is a tremendous invention, allowing the dispursal of information accross the globe nearly instantaniously. I hope that at least some of these posts will be helpful to someone in some way. Evidence for future employers: A resume can only say so much. I hope that this website serves as proof of what I can do to anyone who hapens to be looking. As of the writing, this site is still very new and I am still learning all the options for Hugo and PaperMod (the framework and theme, respectively, that this blog runs off of).\n","date":"1 December 2024","externalUrl":null,"permalink":"/homelab/welcome/","section":"Homelabs","summary":"","title":"Welcome!","type":"homelab"},{"content":"Biography coming soon. For now, here is my resume.\nTo contact me, email: josiah.mckay [at] protonmail.com\n","externalUrl":null,"permalink":"/about/","section":"Josiah","summary":"","title":"","type":"page"},{"content":"","externalUrl":null,"permalink":"/authors/","section":"Authors","summary":"","title":"Authors","type":"authors"},{"content":" Education \u0026amp; Certification # Millersville University\nBachelor of Arts in Entertainment Technology [2021]\nMinor: Music; General Applied Engineering and Technology\nCommunity College of the Air Force\nAssociates in Applied Engineering [2020]\nCertificates # CompTIA A+ CompTIA Network+ CompTIA Security+ AWS Certified Cloud Practitioner Professional Experience # System Administrator # Department of Justice (Contracted via Prism Inc) [February 2025 - Present]\nRadar, Airfield, and Weather Systems # Air National Guard [January 2016 – Present]\nInstalled, maintained, tested, removed, and replaced mission-critical aircraft radio communication systems. Led engaging training sessions for multiple Airmen that increased job knowledge and job performance. Promoted to the rank of Staff Sergeant after completion of Airmen Leadership School and demonstration of competency and integrity. Obtained Secret Clearance. Technology Support Technician # University of South Carolina; College of Arts and Sciences [February 2023 – February 2025]\nTook over primary desktop support for Mathematics and Statistics departments Created inventory and provided continuation of support for over 150 desktop and laptop computers Facilitated Windows 11 migration for over 60 unsupported computers Created and maintained documentation for unique departmental needs Lead the transition of support from email to ServiceNow, decreasing support wait time. Administered existing VMs and physical servers Assisted in transitioning three legacy servers to the University’s VMware environment. Updated existing VM to current security standards. Maintained Active Directory computer and user group records for Mathematics and Statistics departments and modified them as appropriate. Resolved over 150 Desktop Support incidents Tier 1 Network Issues User Support Technician # University of South Carolina; College of Arts and Sciences [July 2021 – February 2023]\nConfigured over eighty new Windows and macOS based desktops and laptops, including setting group policy and joining to a domain. Utilize remote control applications (Bomgar, Windows RDP) to assist off-site personal. Troubleshot and solved over two-hundred incidents both in-person and remotely ranging from resetting credentials to analyzing failed hard drives. Recommended and installed cost-effective and productive hardware and software for end-users. Continually learned about new technologies and improved current skills through various online trainings. Professional Development # Homelab # [2021 - Present]\nSetup and manage TCP/IP networks, including VLANs, DHCP, firewall rules, BIND9 DNS, and Traefik reverse proxy. Install, update, manage, and backup Proxmox nodes running multiple Linux VMs. Configured and managed Docker containers via YAML files. Setup and manage SMB, NFS, and iSCSI network file shares. Research and implement Windows Server 2012 and 2019 with roles for Domain Controller, DNS, DHCP, storage solutions and file sharing, and Group Policy. Use Python, Ansible, and Bash scripting to automate tasks. Expand development knowledge with Golang and Git version control. Research and implement best practice for networking, secure hypervisor and VM deployments, and application configurations via Docker and Kubernetes. Utilized AWS Glacier Deep Archive and Backblaze B2 for backups and DR recovery. Utilized various platforms such as LinkedIn Learning, ITProTV, and YouTube to learn about new technologies and strengthen my current skill set. ","externalUrl":null,"permalink":"/resume/","section":"Josiah","summary":"","title":"Resume","type":"page"},{"content":"","externalUrl":null,"permalink":"/series/","section":"Series","summary":"","title":"Series","type":"series"}]