Jump to content

  •  

  • iBotModz CBox


    Photo

    The 29 hours of downtime.


    • This topic is locked This topic is locked
    17 replies to this topic

    #1 iBotPeaches

    iBotPeaches

      General Grade 6

    • Owner

    • 6,570 posts
    • Joined: 29-July 07
    • Gender:Male
    • Location:Kansas

    Posted 11 January 2011 - 04:31 PM

    I'm going to run you through what happened.
    whoa i don't like to read, skip to the end.


    2 days ago right when I launched this new server it was running incredibly fast. It had a very light weight control panel known as Kloxo. It basically had only the necessities to run a web server, with the addition of security benefits. The site was running incredibly fast, and I finally felt that I had tweak the server perfectly. I'm sitting in SSH, checking over some configurations and I got an email saying there was a SSH login. Which was strange, since I'm the only user. I then get force booted from my SSH session and realize something was up, I could not log back in and I had 0 access to my own server. About 30 minutes later I received an email from some guy saying that Kloxo is worst piece of security ever and said he could take this entire server. His login was jailed to a tiny little directory though, so I didn't think he could do any damage. It was down to asking the DC for a reformat, or arguing with this random guy. 5 minutes later he emailed me again and said in a nutshell, get rid of Kloxo and put a real control panel on there.

    This is all going on as IBM is running smooth as silk. After he sent me my new password, I quickly implemented a private/public key for log in, so now you need to steal my public key to get into SSH. Point being, I looked at the bash_history (what someone types into ssh) and surprised this guy did nothing except taunt me. So now I'm sitting with a free control panel and being forced to upgrade to my personal choice of cPanel. It was 20/month extra with my provider. I said hell no, and bought a direct VPS license from cpanel for $200, but wait. $20 x 12 months = $240, no I did save money :p

    After I purchased that license I knocked the sever offline at around 3pm. I locked the databases to prevent any extra data and began to tar (like RAR) my entire user folder. About 30 minutes later I had all my dB's and files in a .tar.gz file. I then slowly but surely transferred that back to my computer. Took about 45 minutes to transfer. So now I had all my files and reformatted the server to a base 5.4 Centos install. I then began the cPanel install.

    50 minutes later at the end of the install, I get this.

    Securing the /tmp and /var/tmp directories....*** Notice *** No loop module detected
    If the loopback block device is built as a module, try running `modprobe loop` as root via ssh and running this script again.
    If the loopback block device is built into the kernel itself, you can ignore this message.
    *** Notice *** No working loopback device files found. Try running  `modprobe loop` as root via ssh and running this script again.

    I googled and could not figure that out for at least 45 more minutes. I tried reformatting to minimal Centos install on 64 bit, 32 bit, etc. I also tried all those suggestions online, and still no go. I then reformatted for the last time and asked cPanel to install the panel. Took them around 45 minutes to start the install, and then 3 hours to do it. By the looks of the bash_history they encountered the same error, but wrote some strange terminal code that I don't even understand to resolve it.

    So now I'm faced with the initial control panel of WHM and cPanel. Servers are usually setup in days, and the tweaked over weeks. I think I spent 3 hours straight in that stupid WHM area just setting it up. Compared to Kloxo there was 1,903,912 different settings to pick from. All the way from pure-ftp or pro-ftpd, to the minor version of PHP. Once everything was setup I began the slow transfer of this massive 8ish GB tar.gz file. At 123kbps....

    Granted, I figured out why my connection was being throttled and got it back up to 1mbps later on.

    As this slow file is transferring I am performing my optimizations that I saved from our previous server. However, I learned that you cannot just edit my.cnf, or httpd.conf as cPanel will re-generate it with its version. You must edit another file, and save it. Which then modifies the config files. Took me awhile to get a hang of that syntax, as I was continually editing the "cached" version of files. I also was setting up the domains, fixing the DNS and adding the webmails for each of the sites.

    2.5 hours later the transfer is almost done. That stupid large piece of junk tar file finishes. I simple untar it at command line and my entire folder structure is retained (in the wrong place), so I had to do a little cp -a (copy) to get that bugger back to where I wanted. Then came the problem of permissions and chmodding. None of my chmod's were saved, so I had to restore my permissions to allow files to run. Might have noticed those 403 errors?

    Then the worst part comes, apache and virtualhosts with the www extension were failing. For at least 2 hours www.ibotmodz.net would go to a setup page, as ibotmodz.net would go to the site. I broke things and fixed other things for 2 hours until that was fixed. Databases were imported, permissions were restored and alas IBM loaded slowly. My server was back to its un-optimized self due to all the things I did trying to fix the virtualhost bug.

    So I then I went to sleep and left IBM running un-optimize. Woke up a few hours later and began tweaking it to run a bit better. So....if anyone wants to chip in a couple bucks for the cPanel license. You know my paypal :p



    TL;DR = There were problems. They were fixed. bi.

    #2 Quinn

    Quinn

      Colonel Grade 3

    • Global Mods

    • 2,323 posts
    • Joined: 29-August 07
    • Gender:Male
    • Location:Canada

    Posted 11 January 2011 - 05:34 PM

    Read it all, that's harsh

    #3 Randomrocket

    Randomrocket

      Gunnery Sergeant Grade 2

    • VIP

    • 796 posts
    • Joined: 29-January 09
    • Gender:Male
    • Location:Texas

    Posted 11 January 2011 - 07:47 PM

    dang. You miss work or something to do all of this?

    #4 Slidell

    Slidell

      IBM Life Support

    • VIP

    • 2,863 posts
    • Joined: 18-July 08
    • Gender:Male
    • Location:New Orleans

    Posted 11 January 2011 - 07:52 PM

    dang. You miss work or something to do all of this?


    Peaches is an AI. You didn't know that?

    #5 REDSTAR 105

    REDSTAR 105

      Insert Custom Title Here.

    • Donors+

    • 775 posts
    • Joined: 16-April 08
    • Gender:Male
    • Location:You know... over there.

    Posted 11 January 2011 - 09:54 PM

    Posted Image ftw
    but still read halfway through, just glad the site is back up! good job peaches

    Edited by REDSTAR 105, 11 January 2011 - 09:54 PM.


    #6 DarkRuler

    DarkRuler

      Gunnery Sergeant Grade 1

    • Donors

    • 602 posts
    • Joined: 18-August 08
    • Gender:Male

    Posted 12 January 2011 - 12:53 AM

    Shit, that is insane

    #7 gruntmods

    gruntmods

      Commander Grade 2

    • Donors+

    • 1,889 posts
    • Joined: 26-November 08
    • Gender:Male
    • Location:Unknown World

    Posted 12 January 2011 - 05:10 AM

    This is why I dislike private servers, hardly any support.

    #8 fattwam

    fattwam

      General Grade 2

    • Sub-Admins

    • 3,979 posts
    • Joined: 18-August 07
    • Gender:Male

    Posted 12 January 2011 - 05:53 AM

    Glad we are Back =]
    And here's to our growth

    #9 Randomrocket

    Randomrocket

      Gunnery Sergeant Grade 2

    • VIP

    • 796 posts
    • Joined: 29-January 09
    • Gender:Male
    • Location:Texas

    Posted 12 January 2011 - 03:28 PM

    Peaches is an AI. You didn't know that?

    Then how does he sign into websites that use captchas?

    #10 iBotPeaches

    iBotPeaches

      General Grade 6

    • Owner

    • 6,570 posts
    • Joined: 29-July 07
    • Gender:Male
    • Location:Kansas

    Posted 12 January 2011 - 03:48 PM

    AIs can juggle ping pong balls.

    I'm sure they can use captchas.

    #11 Randomrocket

    Randomrocket

      Gunnery Sergeant Grade 2

    • VIP

    • 796 posts
    • Joined: 29-January 09
    • Gender:Male
    • Location:Texas

    Posted 12 January 2011 - 03:53 PM

    AIs can juggle ping pong balls.

    I'm sure they can use captchas.

    lol saw that video. But can they annoy the crap out of someone while they try to play an awesome video game?

    Oh wait.... yeah they can....
    Posted Image

    #12 Quinn

    Quinn

      Colonel Grade 3

    • Global Mods

    • 2,323 posts
    • Joined: 29-August 07
    • Gender:Male
    • Location:Canada

    Posted 13 January 2011 - 02:40 PM

    @Peaches and any other AI

    011000110110010000100000011100100110111101101111011101000000110100001010011000110110100001101101011011110110010000100000001101110011011100110111001
    000000111001001101111011011110111010000001101000010100111001001101101011001000110100101110010001000000111001001101111011011110111010000001101000010
    100110001101101100011001010110000101110010000011010000101001100101011000110110100001101111001000000010001001000111011011110110111101100100011000100
    111100101100101001000000011101000101001001000100000110100001010011100100110010101100010011011110110111101110100000011010000101001101100011011110110
    110001101111011011000110111101101100011011110110110001101111011011000110111101101100011011110110110001101111011011000110111101101100011011110110110
    001101111011011000110111101101100000011010000101001110100011010000110000101110100001000000110001101101111011001000110010100100000011101110110111101
    110101011011000110010001101110001001110111010000100000011001010111011001100101011100100010000001110111011011110111001001101011001000000011101001010
    000

    Edited by Quinn, 13 January 2011 - 02:41 PM.


    #13 iBotPeaches

    iBotPeaches

      General Grade 6

    • Owner

    • 6,570 posts
    • Joined: 29-July 07
    • Gender:Male
    • Location:Kansas

    Posted 13 January 2011 - 03:51 PM

    Yep, ain't no root folder.

    #14 Lockon

    Lockon

      "I never asked for this"

    • VIP

    • 1,896 posts
    • Joined: 17-June 08
    • Gender:Male
    • Location:N.Ireland

    Posted 14 January 2011 - 01:29 AM

    Quinn you forgot "01100101" Which is an 'e' Thats why it doesn't work =D

    #15 Twis7eD

    Twis7eD

      Private Grade 2

    • Members+

    • 89 posts
    • Joined: 19-December 09

    Posted 14 January 2011 - 03:31 PM

    I hope you didn't manually Chmod all your files back to the correct permissions. That would've been the biggest pain in the ass ever.

    #16 iBotPeaches

    iBotPeaches

      General Grade 6

    • Owner

    • 6,570 posts
    • Joined: 29-July 07
    • Gender:Male
    • Location:Kansas

    Posted 14 January 2011 - 04:09 PM

    I chmodded them all to a safe powerless number, then manually went back and changed those that wanted 777, and 750 and 444 etc.

    #17 gruntmods

    gruntmods

      Commander Grade 2

    • Donors+

    • 1,889 posts
    • Joined: 26-November 08
    • Gender:Male
    • Location:Unknown World

    Posted 15 January 2011 - 09:39 AM

    I chmodded them all to a safe powerless number, then manually went back and changed those that wanted 777, and 750 and 444 etc.

    its not that big of a pain in the ass, but it does take awhile.

    #18 Twis7eD

    Twis7eD

      Private Grade 2

    • Members+

    • 89 posts
    • Joined: 19-December 09

    Posted 15 January 2011 - 09:32 PM

    ^Well, yeah, that's pretty much what I meant by a pain in the ass. I feel bad just for that part, but in addition to everything else, that really just sucks.