We should ditch NGINX

We really need to migrate off of NGINX. The news of their commercial offering earlier this year wasn’t well received but the magnitude of the problem wasn’t clear to me until earlier today. We’re working on centralizing logging (what a concept), not just as a best practice for issue resolution and health monitoring, but to reduce the write load on our beleaguered old NetApp at One Wilshire.

You see, Black Friday is coming up and we’re already observing latency spikes during the 8am sales event. Cyber Monday is one flash sale event every hour; I don’t have to do the math there. We’re virtually guaranteed to crush the NetApp with this onslaught. Even though our CEO is a great guy who understands our saddling of technical debt, I can’t help but get a sinking feeling when the website takes a fat dump and he moseys on down from the 3rd floor to ask “how we’re doing” and “how can we mitigate future issues.”

So back to NGINX. Our noble HTTP server will generate, in aggregate, around 50M events a day. These all get flushed to disk from hundreds of VMs, whose hosts in turn flush their images to the NetApp. Getting these NGINX log events into a remote server is easy with rsyslog. But preventing the log events from writing to disk in the first place? Fucking impossible, unless you a) don’t want logs or b) pony up for the commercial offering that patches direct syslog into the NGINX logging engine. Yep, they took a feature that Apache httpd has had literally forever and put it behind a pay wall.

Oh sure, there are patches that’ll add in that ignominious feature. It’s trivial to pull down the NGINX source, pull down the patch, patch the source, test the build, create a spec file, test the RPM, add it into the yum repository, and stage an untested major update to every web server just before the holiday rush. It’s just that I shouldn’t have to do any of this shit. Tested, reliable packages are the entire point of upstream package sources like EPEL. Patching NGINX in-house is a huge step back.

So what next? In the short term we’ll probably do something hackish like mounting /var/log/nginx on a ram disk and get some serious logrotate action going. It’ll eliminate those dastardly writes and get us out of the woods. As a proper solution, httpd is sounding quite nice all of a sudden. We’re using php-fpm anyway; the performance difference between nginx and httpd in this scenario is negligible.

I don’t blame NGINX for wanting money in exchange for what is admittedly amazing software. I just don’t want any part of it. This is just the first feature of presumably many that the FOSS NGINX distribution will lack compared to the commercial offering. In short, this is Oracle and MySQL all over again. I wonder if we’ll see the same apparent exodus from NGINX as we have from MySQL? Who will pick up the mantle and actively develop the next game-changing FOSS web server?

Fork it.

42 thoughts on “We should ditch NGINX

  1. Pingback: We Should Ditch Nginx | Boardmad

      1. voretaq7

        It amazes me how you guys make the inductive leap from our CEO is a great guy who understands our saddling of technical debt to Let’s go for the technical debt ceiling by forking a major system component, applying our own patches, and increasing our test burden exponentially for every push!.

        I’m just going to go ahead and assume y’all work for companies with thousand-person shops that can do the integration and testing without breaking a sweat, because the alternative would require me to be insulting and belittling.

        Reply
  2. Pingback: We Should Ditch Nginx | Rocketboom

  3. Pingback: We Should Ditch Nginx | Enjoying The Moment

  4. Travis

    I know I’m biased because I love splunk but I think it could really help you out here in all sorts of ways.
    And my opinion is you should switch back to apache given the performance differential is so marginal.

    Reply
  5. l8rs

    Really? I love nginx. This bit of negative news on its use in high scale production systems is most unfortunate…

    Reply
  6. Manu Mukerji

    I am wondering if you can elaborate a little on the bottleneck ? I have used nginx before for logging over a billion hits/day..

    In out setup we had a bunch of c1.mediums that did 5000/qps of logging then we rotated them to s3…

    -Manu

    Reply
  7. Mark V

    But that’s the whole point of commercial license: you’re a company, it’s easy and pretty much affordable (actually, I’d guess ‘negligible’ is a better word) for you to pay for the thing you need and not play Slackware on a production system. Not a student, not a bunch of hipster kids — a company. Feel the difference.

    Let’s be honest, how much do you, as a business, spend on Cola and sandwiches in the fridge? Nginx license will be comparable even for a very small team, and you’re talking “from the 3rd floor”-size company.

    But no, you don’t want to pay a negligible sum, AND you don’t want to work around the issue; you want someone else to do it for free.

    “Fork it”, he said, “for I am greedy and unwilling to support developers that build systems my stuff relies on”. Or something like that.

    Reply
    1. XII

      While I agree with the criticism of not supporting companies that provide the tools on which your own company is built, I think there’s also a valid point to be made about generally being weary of open-source software with crippled basic functionality. An open-source database with (non-free) enterprise management tools can make sense, but deliberately and spitefully refusing basic features is rather sad – yes, I’m an engineer and not a marketing person.

      Reply
    2. Craig

      Oh, do shut up with your silly, “you’d pay less for a fridge of sandwiches” nonsense. That’s used car salesman talk.

      Reply
  8. Clemens Vasters

    Let’s see. You talk of “Black Friday” and “Cyber Monday”. That tells me that you’re running a commercial website selling stuff to people. Very likely following the centuries-old capitalist practice of selling whatever you’re selling for more than you bought it for. Yet, software that enables you doing this business at scale must be free as in beer. That’s deeply flawed thinking. Here’s a page where companies like yours ought to be listed: http://nginx.org/en/donation.html

    Reply
  9. nick

    Hey! Are you referring to One Wilshire in Downtown Los Angeles? If so, we should connect. I’m current an entrepreneur in an accelerator called StartEngine. I’m new to the area and would love to meet more LA hackers.

    Reply
  10. James

    To summarise – entitled whiny bitch doesn’t want to pay for software which is generating business value for him, so wants someone else to fork it for him and work for free so that features are available at no charge to him so he doesn’t look stupid in front of his boss for not understanding system bottlenecks in his architecture until it’s too late.

    Sounds like this feature would be a simple patch that a moderately knowledgeable C programmer could make. Open source isn’t just a free-for-all. Quality software costs money to create – as far as is evident from this blog you’ve contributed nothing either to OSS, or indeed to humanity.

    “Fork you”.

    Reply
  11. Pingback: Kredit

  12. Tomeu Vizoso

    The author is also using the English language, should he/she pay for it as well?

    I fail to see why a company should be getting money again and again for the same piece of code, as it was written just once.

    Reply
    1. coward

      Software development is an ongoing process, involving patches, bugfixes, updates, new features, testing before rolling out to the userbase and plenty of stuff i cant think right now. Ongoing process, also involve ongoing costs. If they want their software to stay alive and up-to-date, some sort of income should be guaranteed.
      Just my 0.02

      Reply
  13. Joe

    Can you add an extra spindle devoted just to logs on the nginx servers? My guess as to why nginx hasn’t prioritized getting that working is it’s probably not the best idea to send all logs over the network without keeping them somewhere local first. Something happens with that remote server or the network connection to it you could have a couple problems.

    First, depending on how the tcp connection for the syslog is built in nginx, you could starve yourself of connections or memory. Second, you could create a logging black hole.

    Flushing to disk really is, in my opinion, the safest route. I’d treat it more as a io engineering problem than change your web stack.

    Reply
    1. joel Post author

      This is awesome! Thanks for taking the time! We’re going to pow-wow today and come up with a strategy, this is first on the list.

      Reply
  14. Pingback: Are Open Source Developers Too Demanding? - IT Clips

  15. Pingback: Are Open Source Developers Too Demanding? | SEO Facts

  16. Pingback: Are Open Source Developers Too Demanding? | Matias Vangsnes

  17. Pingback: Are Open Source Developers Too Demanding? - Techno Yard and Online Store - Tech Clot- Techno Yard and Online Store – Tech Clot

  18. Nero

    Why not try to buffer the logs?

    You can use something like buffer=32k or something to reduce the no. of writes.

    Reply
  19. sirshitfaced

    DevOp maintains big e-commerce site.
    DevOp didn’t bother to run tests and evaluate what software stack best fits their needs.
    DevOp is handed solution to problem he didn’t think of.
    DevOp complains that he has to spend money for the solution that would save his skin.
    Op should be taken out back, shot twice and sued for fucking up at such an easy job.

    Reply
  20. Pingback: Are Open Source Developers Too Demanding?

  21. Dg

    If your for-profit website generates 50M events/day you can easily afford a commercial license. Pay up you cheap bastards.

    Reply
  22. zf

    apache 2.4 with event mpm working well for us, welcome to our truely open world, the apache ,will save you

    Reply
  23. Pingback: Are Open Source Developers Too Demanding? | DIGIZENS

  24. Pingback: Are Open Source Developers Too Demanding? « DevelopersArena.com

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>