The Official Status Update Thread: Part 91 - Snowmageddon 2 (The White Death)

[Frosty]

Well, I have been called into duty. No heat at a neighbors house down the street. Guess I have to get dressed and go out "THERE"

Be safe papa, it's hip breaking cold out

 

[mom45]

I built a fire in the upstairs wood stove so maybe we can warm things up a little around here. Gonna make a pot of chili....

 

[MrsGungho]

braved the elements and made it to speedway for a fountain diet pepsi. roads are packed ice but passable. The alley was a mess to get out of, but I did it.
Saw the boss was at work so now I am waiting for a call to go to work.
I know I can make it now.

 

[Frosty]

Krogers looked to have 3 employees on staff today, family video had 1, two gas stations open here but nobody in them, not much moving in this little town...

 

[churchmouse]

Be safe papa, it's hip breaking cold out

You are truly a funny man........

Oh poop, it is freaking cold outside.

 

[churchmouse]

I built a fire in the upstairs wood stove so maybe we can warm things up a little around here. Gonna make a pot of chili....

Daughter made some great chili last night. I saved just enough to make me an omelet tomorrow. MMmmmmm....chili Omelet

 

[mom45]

Daughter made some great chili last night. I saved just enough to make me an omelet tomorrow. MMmmmmm....chili Omelet

Still trying to convince stubborn daughter to stay home because she is not supposed to be on the roads and that she will NOT get fired if she doesn't go to work. I have no idea where she gets this stubbornness from.....no idea at all.......

 

[MikeDVB]

Well... The server has 64 GB of RAM and for optimal performance it uses a good portion of that for caching. For example let's say you want to write a 2GB file to the disks but the disks are busy doing other things - we can either hold up the other processes and write our data, hold on writing our data and let everything else run, or cache the data and write it as soon as the disks are available/not busy.

The first option would cause everything to halt but that one write - not good in a shared environment. The second would cause whatever process was trying to write to halt until there was disk write available - not good in a shared environment either because it could happen to everybody at any time. The third option is what *should* happen but the 'slab_cache' most notably the dentry cache is building up to huge levels [it never should go above 2~4gb and it's going up to 30~40gb].

In short - the system is cutting off it's nose to save it's face and it's not something I can prevent.

The funny part is that we use SSD Cached I/O to prevent issues like these - but when our backup system is running and this issue happens it compounds to the point of things basically coming to a halt across the board.

A few options:
1. Disable our R1Soft backups - don't want to do this but probably will have to.
2. Deal with people complaining and explain it's not something within our control but we're working with our software vendor - less than ideal.
3. Disable R1Soft and use an alternative backup system - this is what we're doing but it's much more wasteful and much less robust - but you gotta do what you gotta do.

What's most irritating is that if I weren't a decent server admin I'd never have even caught this - it would have just been random unexplained lack of performance.

 

[SEIndSAM]

Yep, getting colder instead of warmer.

 

[churchmouse]

Well... The server has 64 GB of RAM and for optimal performance it uses a good portion of that for caching. For example let's say you want to write a 2GB file to the disks but the disks are busy doing other things - we can either hold up the other processes and write our data, hold on writing our data and let everything else run, or cache the data and write it as soon as the disks are available/not busy.

The first option would cause everything to halt but that one write - not good in a shared environment. The second would cause whatever process was trying to write to halt until there was disk write available - not good in a shared environment either because it could happen to everybody at any time. The third option is what *should* happen but the 'slab_cache' most notably the dentry cache is building up to huge levels [it never should go above 2~4gb and it's going up to 30~40gb].

In short - the system is cutting off it's nose to save it's face and it's not something I can prevent.

The funny part is that we use SSD Cached I/O to prevent issues like these - but when our backup system is running and this issue happens it compounds to the point of things basically coming to a halt across the board.

A few options:
1. Disable our R1Soft backups - don't want to do this but probably will have to.
2. Deal with people complaining and explain it's not something within our control but we're working with our software vendor - less than ideal.
3. Disable R1Soft and use an alternative backup system - this is what we're doing but it's much more wasteful and much less robust - but you gotta do what you gotta do.

What's most irritating is that if I weren't a decent server admin I'd never have even caught this - it would have just been random unexplained lack of performance.

Now that makes sense to us dummy's....at least to me.....

 

[Frosty]

You are truly a funny man........

Oh poop, it is freaking cold outside.

funny haha or funny likes boys funny it is cold, I had on insulated rubber boots and my toes were still so cold I could hardly walk when we got done!

 

[Frosty]

Now that makes sense to us dummy's....at least to me.....

Nope, I'm still lost. Of course my knowledge of computers is limited to fixing it with control alt delete or the escape key, if those don't fix it I'm out of ideas!

 

[churchmouse]

funny haha or funny likes boys funny it is cold, I had on insulated rubber boots and my toes were still so cold I could hardly walk when we got done!

Well funny ha ha of course. Now I am not so sure about a few others in here.....

 

[churchmouse]

Nope, I'm still lost. Of course my knowledge of computers is limited to fixing it with control alt delete or the escape key, if those don't fix it I'm out of ideas!

I do the regular in boox maint. on my lap top but past that I have a son and daughter.....they can do it.

 

[chocktaw2]

Well funny ha ha of course. Now I am not so sure about a few others in here.....

You called

Howdy CM, good afternoon

 

[SEIndSAM]

I'm bored, ready to get back to work and things back to normal.

 

[chocktaw2]

I'm bored, ready to get back to work and things back to normal.

Hi Sam, new normal, or the old normal?

 

[SEIndSAM]

Both. I need to get out of the house. Bored to tears.

 

[Frosty]

Both. I need to get out of the house. Bored to tears.

Stop talking like that, I'm waiting on the call for another snowday!

 

[chocktaw2]

It's fricking cold out there. Did your Bengals win yesterday?