Thinking about processes on the service level

Recently, I was tasked with improving site performance and page load speed by refining our server architecture at one of my positions. I had given this some thought in the past, but this gave me an opportunity to reconsider the way I look at infrastructure. In one of the projects I’m working on, I had to think a lot about purpose differentiation – having different processes on different servers that communicate between each other as opposed to having one big behemoth of a program that requires a monster server to run on. I came up with a solution that I’m very happy with, not only because it lets us troubleshoot really well, but also because it allows for easy scaling out of any bottlenecks and maximizing server efficiency. You can think of it like service-oriented architecture, but in the sense of server roles instead of just application services; actually, working with SOA principles in another one of my projects originally gave me the idea for what I did here.

Starting Out

To start thinking about this, let’s talk a little about server architecture. If you’re just starting out in the Linux world, or if you’re not working on a large scale, it’s very possible that you’re going to be running a LAMP, LEMP or similar stack, meaning a web server, PHP, and a database running on one Linux server.

Now, this is all well and good, especially when you’re operating on a smaller scale. In fact, this is still how I run my personal sites – since I’ve only got 10 or so, a unified server works just fine. However, running a larger-scale infrastructure requires something slightly more complex. To that end, let’s think a bit about that discrete-service concept I touched on at the beginning of the post, and talk about how it applies to the first step that most sysadmins will take when scaling out their operation.

When a basic LAMP stack starts to run into performance issues and no amount of tuning will help, most people will spin off the MySQL server into its own discrete server. This tends to help with performance quite a bit, because MySQL and other database providers can be pretty memory intensive, making what process to spin off first an easy decision. Once you’ve done that, you can start doing fun stuff like MySQL clustering to improve database performance as well. I think of this as the first step – now, you’re treating MySQL as a discrete service which you can scale without having to touch your web server.

Scaling Further

The problem arises when you outscale that. At some point, the rest of your stack is going to run into a problem, especially if you (as I did) start implementing more technology like Varnish or Redis in addition to everything else. Sure, you can scale the whole box – but what problem does that really solve? You’re duplicating a lot or processes that don’t need to be duplicated to give more juice to the one that’s slowing you down – plus, it’s a lot harder to figure out exactly which process is bottlenecking you.

Instead of going down the path of infinite horizontal scaling for a full stack, let’s think about how we could apply the discrete service philosophy to our remaining processes. In most cases, the caching stuff like Varnish or Redis is going to be very easy to spin off, so we may as well do that – leaving us with just Apache and PHP processing local files.

There’s still two more processes that we can make discrete, though. First, we can move the storage onto another server. I’m using DRBD in my setup (thanks to Toki Winter for a wonderful writeup on this), which allows for high-availability storage accessible over a network. If you’d rather use a SAN, that’s also a great solution.

We’ve still got one last piece, though, and that’s the piece that I just recently spun off – and that’s PHP. In a web environment, your web server really isn’t what’s causing the load in most cases. Rather, it’s PHP which is eating your resources. With that in mind, it’s worth looking into spinning off your PHP process into a FPM cluster. (Thanks have to go out to Jamie Alquiza‘s incomparable writeup on spinning off FPM. He mentioned using rsync and local file storage on the PHP-FPM cluster, which may be another performance boost, but I haven’t tested it versus DRBD.).

Optimizing Your Environment

Now that we’ve got all of our different processes on their own servers, it becomes trivial to find performance bottlenecks. Site running slow? Check your Zabbix dashboard (shameless plug!) and see what server’s getting pegged. From there, it’s a simple task to spin up another server and stick everything behind a HAProxy load balancer (preferably set up with high availability using heartbeat or pacemaker/corosync, depending on your environment) – and voilà, instant bottleneck reduction! Not only that, but this allows you to more efficiently allocate your resources in a virtualized environment. If you spin off all your services and find that only certain servers are really hitting their upper limits on CPU or memory, you can reallocate excess resources directly to the processes that need it.

Now, this is all something where your mileage might vary, and solutions might vary too. For example, instead of just spinning off MySQL, I’ve been working on getting a Galera solution working in our development environment. In other cases, problems might be solved via tuning; maybe a PHP memory limit needs to be adjusted, or Apache needs to be tuned. Finally, if you’re already pressed for system resources, spinning off into more systems and adding slightly more OS overhead may not be the best option for you. In a typical situation, though, I think it’s possible to really increase your efficiency with a philosophy like this one.

How to install Symantec Backup Exec Agent on CentOS/RHEL 7 Minimal

Recently, I was helping evaluate new long-term backup options to replace Microsoft’s DPM due to their lack of support for Linux, and one of the options that we were looking at was Symantec Backup Exec 2014. Unfortunately, if you look at their software compatibility list, you’ll notice that they do support Red Hat 7, but with a very important caveat – you can’t install as minimal.  This was kind of a problem for us, since we had built all of our servers using a minimal image, and I wasn’t about to rebuild twenty-plus servers just for a backup solution.

Fortunately, the fix was pretty easy. After installing a base non-minimal image, we compared the list of packages, and found some differences. After testing a couple of packages, we found that the one that mattered to the Backup Exec agent installer was the net-tools package, which contains the ifconfig command that the agent installer needs.

After installing this via yum, you’ll be able to run the Backup Exec installer. However, the SymSnap driver install will still fail. I wasn’t able to figure out a way to solve this, since it failed on both the full install and the minimal install, so if anyone has some guidance for that part of the installation I’d love to hear your solutions!

2014 – Year in Review

What an exciting year it was for me! I haven’t had time to really sit down and write a retrospective, but I figure I should probably get around to it before 2015 gets too far off the ground.

It was certainly a busy year. Back in May, I proposed to my wonderful fianceé Sarah, and since then there’s been a lot of wedding planning happening. It’s kept me on my toes trying to make sure that everything’s in order and ready to go for that! Both Sarah and I are extremely excited for the wedding, which is coming up this June.

I spent a lot of the early and middle parts of the year writing a lot of code. Between relaunching my personal site, developing MyLeaf, and working on all of the fun projects like MaltSwap and zabbixweb, I didn’t have a chance to take a step back and reevaluate what exactly my focus was. Fortunately, I was also doing a lot of PHP-related stuff at work, and that was where my wake-up call finally came from. I took the opportunity to refocus and work more on the systems side of things for a while. That said, though, I still want to keep doing development – just more casually than what I was doing before.

My move back into systems led to a lot of opportunities, not just in my main job but also elsewhere. I recently took on another systems job with a local startup, and have been really involved in designing continuous integration and server infrastructure. It’s been a lot of fun, and I’m looking forward to being part of that for a while yet.

That whole thing has really gotten me to the point where I’m passionate about systems again. There was quite some time where I was a lot more interested in code, and it’s good to get back to really enjoying servers and infrastructure. Not to say that my time coding was not time well spent – not only was I pretty productive, but I also learned a lot that I’ve been able to translate back into the systems side of things, especially recently writing some shell scripts to help move us towards true continuous integration.

As many of my readers know, I’ve been reading a lot for a very long time. I’ve made a point recently of making sure I take some time every day to get in some book time, and it’s been really a great way to wind down before bed. Similarly, I’ve been getting on the exercise bike every day as well. It’s all about forming the habit, really: once I got into the routine of getting home and hopping on the bike immediately rather than getting home and hopping on the computer immediately, it became really easy, and I’ve been feeling so much better in general since I started. That bike’s been great value-wise for me, especially considering all it cost me was fifty bucks and a drive out to Grand Island.

Recently, there’s been a nice spurt of development on zabbixweb! If you have time to check it out or throw in some improvements, that’d be pretty awesome.

Anyway, other than that, things haven’t been too eventful. Sarah’s back down for a visit now, which is very exciting. We’re doing a lot of wedding planning, including some shopping this month for various important things. Should be a blast.

I’m hoping to have some news on MaltSwap as well as another new project coming up in the next month. I’ll be sure to keep you all posted.

Quick thanks

Hey, some of you may have noticed the blog was down over the weekend. Thanks to the person who let me know – not sure who you are, but you’re awesome.

It shouldn’t go down again. I was missing a DNS entry after migrating to my new (awesome) DNS provider, Hover, and just had to add it back in.

I’m working on another blog post and am hoping to have it up this week sometime.

Installing Varnish 3 on CentOS 7

Oddly, there isn’t a lot of information on installing Varnish 3 on CentOS 7. There’s well-written instructions on the Varnish site for CentOS 5/6, but for 7, it seems like you’re stuck with Varnish 4 (from the epel-release repository), especially if you’ve searched around and found forum threads like this one.

I did some digging, though, and it turns out Varnish 3 actually does have packages for CentOS 7 – there just isn’t any documentation on how to install them. I played around a little and found something that worked, though. So far, I haven’t experienced any problems installing via the following:

yum install gcc
 rpm -i https://repo.varnish-cache.org/redhat/varnish-3.0/el7/x86_64/varnish/varnish-libs-3.0.6-1.el7.centos.x86_64.rpm
 rpm -i https://repo.varnish-cache.org/redhat/varnish-3.0/el7/x86_64/varnish/varnish-libs-devel-3.0.6-1.el7.centos.x86_64.rpm
 rpm -i https://repo.varnish-cache.org/redhat/varnish-3.0/el7/x86_64/varnish/varnish-3.0.6-1.el7.centos.x86_64.rpm

You do need to install the packages in this order, as Varnish depends on having gcc and the libraries in place.

You may need to adjust the Varnish config and point the pidfile to varnishd.pid instead of varnish.pid in /etc/rc.d/init.d/varnish. If you do, remember to run systemctl daemon-reload and kill all of the active varnish processes before starting the Varnish service again.

Excitement and anxiety

There’s been a lot of excitement in my life recently, especially when it comes to my upcoming wedding. Between favors, venue, food, music, the service itself, and everything else, it’s been a hectic few months of planning, but it’s been a lot of fun too. A lot of the prep’s done now, though, so getting a short bit of time to relax has been nice and hopefully will allow me to start working on some other projects that I’m looking forward to getting to grips with.

One other thing that I’m hoping to find some more time for is playing my violin again. I haven’t touched it in a while now, which I’m a little sad about – but there just hasn’t been time. At the very least, though, I have put together a nice list of things that I want to try when I can find the time – first on that list is the Mozart Concerto in A major, which I’ve listened to a lot recently. I’m really enamored of the third movement.

I know I don’t talk about music a lot on this blog, and I’m not honestly sure why I don’t – I absolutely love music in general, both listening to it and playing it. I think that might be something you see more here in the future, along with the tech stuff.

Anyway, the title of this post was “Excitement and anxiety”, and I swear there’s a good reason for that. I submitted Sarah’s I-129F back in September, and now that we’re coming up on November, I’ve watched the processing times at the relevant visa center (Texas) very closely to see when I’ll be receiving communication – and from the looks of it, it could be pretty soon. The reason this makes me a bit anxious is not because of the possibility of it being declined – something like 99.5% of applications get approved, so no worries there, really – but rather that there’s a lot of stuff that has to be done as soon as I get that notification. Letters to get written, documentation to get put together…yeah. Busy times. Definitely worth it at the end of the day, though.

Anyway, it’s going to be a very busy next couple of months. I’m looking forward to it, but at the same time I’m just going to have to make sure I’m not overwhelmed by all the stuff coming in. Either way, I’ll keep you all posted.

Zabbix web interface up on GitHub

Just a quick note that I’ve added the Zabbix frontend to a GitHub repository. There’s been a lot of interest in this frontend over the last week or so, and to that end I figured it’d be nice to have a place where we could all get together and work on improving it. Feel free to make pull requests and add issues, and we can have a way better Zabbix monitoring solution than what I originally released!

I’m really excited for a lot of the projects coming up for me in the near future. I’m currently starting to learn Laravel, and I’ll probably blog about that pretty soon. I haven’t forgotten about the other posts that I’ve promised, though!