How To Change the Hydraulic Brake Pads on Your Bicycle (and properly bleed the lines)

This procedure is pretty generic and should work with most hydraulic brakes. One of the goals of this post was to use as few photos as possible.

It’s important that you don’t skip any steps here.  Skipping a step or not following the directions exactly may result in spilling hydraulic fluid, dropping the hydraulic reservoir by accident and splashing it on your friend’s shoes, or worse, getting hydraulic fluid on your pads.

There are two reasons you might want to change your pads:

  1. They got contaminated with oil of some type. You will know they got contaminated because they will make a very loud SQUAWK when you use them
  2. They are worn out.  You will know they are beginning to be too thin because they will start to make an unpleasant sound when you brake or may begin to vibrate. New pads are 8-12 mm thick. You’ll want to check with the manufacturer to see when you should replace them.

What you need:

  1. Either a really robust set of Allen and Torx wrenches, or one of the kits they sell for bikes.  These will usually have everything you need
  2. A small glass or a large shot glass, something you can place the reservoir into and is a tight enough fit to keep it from falling over.
  3. Some sort of container, like a shampoo bottle or something (cleaned) that you can use to dispose the fluid.  If you don’t have a clumsy friend around to accidentally spill your fluid on the ground, here is a discussion of several interesting methods of disposal/recycling. I prefer the recycling approach.
  4. A SPRAY bottle of 95% Isopropyl alcohol – they sell them at Walgreens in the first aid area.
  5. A roll of paper towels
  6. A small pair of flat ended pliers. A set like this will serve you well for many years. https://www.amazon.com/CRAFTSMAN-Pliers-6-Piece-Pouch-CMHT81716/dp/B07QL3CLRS/ref=sr_1_8?dchild=1&keywords=pliers&qid=1624834837&sr=8-8

The PROCEDURE

  1. Remove both wheels and support the bike somehow. I used a large plastic bin and a friend to support the front fork and keep the bike from falling on my head.  We supported the bike at the center with an old paint can. The plastic bin also makes a convenient bench.  Find the smallest chair you can, or a log, or maybe your spouse’s favorite footstool, so you have something to sit on.
  2. Make certain you read through this procedure ENTIRELY and check as you read that you have a tool that will fit each bolt head.
  3. Buy one of these.  It comes with more than enough oil for the job for two brakes, even if you accidentally spill the entire reservoir on your friend because you have butter fingers. https://www.amazon.com/dp/B01LWN4LSD/ref=cm_sw_em_r_mt_dp_VFFBE7C0QE5D46QR349J?_encoding=UTF8&psc=1
  4. To avoid contaminating the brake pads, first we are going to completely drain the fluid from the system.
    1. The brake assembly down by the disk, located usually on the left side of the fork a the center of the wheel, is called the “caliper” and it comes apart into two pieces.  Observe how the hydraulic line from the lever housing goes into only one side of the caliper. This will be the “inside” caliper.  I will call the other one the “outside caliper.”
    2. On the bottom of the outside brake caliper, locate a small, plastic cap.  Pop the cap off, holding it securely.  There should be a little ring holding it to the bleed valve nipple, but they do break sometimes.  You don’t want to lose this because dirt can get compacted in the nipple and, if you perform this procedure in the future after having lost that cap, you will surely clog your lines.
    3. In the kit you bought, locate the flat, elliptical tool that has notches in both ends. The tool is a flat piece of plastic, shaped roughly like an American football. Clip it to the end of the clear tube. One of the notches may fit your nipple better than the other. Use the end that fits most snugly.
    4. Test that you can seat it all the way down onto the nipple. This is super important. It must be ALL THE WAY SEATED. Just leave it there like that. It should stay, all on its own.  If it doesn’t, you screwed up.  Try again with the other side or get some help because you might not have the chops to do this.
    5. Using your pinky and thumb, attach one of the knobby things with the flanged nipples to the syringe.  It’s plastic, so be careful not to cross thread! The rule of thumb with plastic parts is don’t go past pinky tight!

  1. Fully depress the syringe plunger
    1. Attach the syringe to the tube. 
    2. On the outside caliper, above the nipple, is a screw.  Loosen it two or three turns
    3. On the brake lever housing you will see a circular, flat, screw head. It receives one of the smaller Allen wrenches. Completely remove it and put it in a bowl that you borrow from your wife’s pantry without telling her.
    4. Pull the plunger, drawing all of the fluid into it.  If there is no fluid and it is hard to pull, then the valve didn’t open.  Maybe you turned the wrong screw.
    5. Once you have successfully drawn out all the fluid, tighten the valve screw and, while pulling gently on the plunger, remove the hose from the nipple and dispose of the fluid into the shampoo bottle. Thoroughly clean the syringe and hose using the isopropyl alcohol. Place the tip of the spray bottle of isopropyl directly up to the end of the hose and flush the tube.

Congrats! You have successfully drained the system for the front brakes.

Replacing the brake pads

What you will need:

  1. You need to have first drained the system. If you skipped the first half of this checklist, go back and do it. Seriously. It is important that you drain the system or you will almost certainly contaminate the pads and make a fucking mess.
  2. A pair of helping hands OR some rubber bands

The Procedure

  1. The pads are held in by the geometry of the calipers and also a cotter pin, or maybe a regular bolt. Remove the cotter pin (or the single bolt). If it is a cotter pin, bend the end straight with a mini pair of flat nosed pliers.
  2. The caliper is held together by two bolts.  They will likely be Torx. Remove them both and place them in the bowl. The assembly should fall apart in your hand. Note that hydraulic fluid does not leak out all over the place. This is because you were smart, and drained it all.
  3. There may be some small tiny amount of leakage. wipe down the area, as follows: Spray a towel with isopropyl alcohol and clean the entire area thoroughly.  If the pads were contaminated, take a family member’s toothbrush (clean it thoroughly with hot water and then alcohol) and spray the pads and get them nice and soaked.  Brush them vigorously with the brush. Wipe with the towel.  Do this until the towel comes away clean.
  4. Place the new pads into the caliper in such a way as to match the geometry therein. The metal clip AND the two calipers should be securable with the cotter pin.  
  5. Have a friend hold the caliper together while you insert the pin. Once the pin is in, and through all three parts (the two pads and the metal brace), screw them together.
  6. There is a flat, plastic, thin block. This is a spacer that is slightly thicker than the disk. Insert it between the pads.Use a rubber band to hold it in place if needed.
  7. Attach the reservoir to the hydraulic chamber on the top of the brake lever body. Thread it carefully, pinky tight. Ensure the stopper is inserted.
  8. Fill the reservoir as high as you can. 
  9. Insert the tip of the syringe into the reservoir and draw as much fluid into it as you can
  10. Connect the hose to the nipple on the outside caliper, ensuring a snug fit.
  11. Connect the other end to the syringe.
  12. Open the bleed valve a couple turns.
  13. Ask your friend to pour just a quarter inch or a centimeter or so of hydraulic fluid into the reservoir.  This will keep it from sputtering and spraying in his face when you press the syringe in the next steps. It will also allow him to redeem himself in the event he dropped it on you before.
  14. Have your friend remove the stopper and place it in the bowl.
  15. Depress the plunger, holding it higher than the caliper and pointing toward the ground
  16. Have your friend tell you when it stops bubbling. 
  17. Repeat the drawing and pushing of all of the fluid, without ever fully emptying the reservoir, two or three times or until all air is out of the tube between the syringe and the caliper, and there are no more bubbles. Go slowly! Wait a few moments between draws to give bubbles a chance to work their way out. The pulling and drawing of the plunger should work to purge the air completely.
  18. When completed, reinsert the stopper.
  19. Close the bleed valve tightly. 
  20. Remove the reservoir, dumping the excess into the recycling. It may have little bits in it from the system, so don’t reuse it. Place the reservoir into the shot or juice glass to hold it because you will need to use it again to do the rear brakes.
  21. Insert and tighten the brake lever housing hydraulic cap screw
  22. Remove the yellow spacer
  23. Repeat the entire procedure on the rear brakes. They will be very similar.
  24. Ensure the brake disks are clean – use the alcohol and towels to ensure their cleanliness.
  25. Reattach your wheels. Brake check by walking the bike and depressing the brake lever. Next, test by riding slowly.  If you hear a squawk, stop and immediately remove the wheel.
  26. Spray the pads with the alcohol. Using a folded paper towel, gently drag it through the opening until you are sure it is clean. 
  27. If you still hear squawking, repeat the entire process.  Practice makes perfect!

You are awesome! You have successfully replaced your brake pads and bled your brake lines! 

The New Bird Collective

This is the officially unofficial naming convention for bird collectives, hereby and forthwith known as “The New Bird Collective” and is wholly different and intended as a replacement for the old, archaic, “Old Bird Collective” which is as seeped in mystery as it is of shrouded (possibly questionable) origin. Some old bird collective nouns will occasionally appear here because, let’s face it, some of them aren’t so bad.

We (this is the royal “we,” as we are related to the Royal House of Orange and are therefor so entitled) do hereby declare that the unofficial designations of collective names be wholly owned by members of the Facebook Bird Misidentification Page 

The first term listed is the generic term that can be used for any collective in any state: perched, flying, floating, sinking, served as a dish, floating, diving, swimming, etc.  You can, however, truly show off your mastery of this list by knowing the list more thoroughly and knowing the most applicable term for any situation.

The New Bird Collective:

Boogey of birds

swarm, reaction of bee-eaters

taste of bitterns

  • truth of bitterns

bakery of blackbirds

  • pie of blackbirds (exactly four and twenty)
  • pies of blackbirds (some exact multiple of four and twenty blackbirds)
  • piefull of blackbirds (far too many blackbirds to count, like during full migration)
  • montrose attack of red-winged blackbirds
  • r2d2 of bobolinks

happiness, mountain of bluebirds

A golf course of bobolinks

bouquet of bulbuls

bazaar of buzzards

  • buffet of buzzards
  • waiting of buzzards (because wait! a buzzard is not really a species…)

barrel of bobolinks

A frederick’s of boobies

  • pageant of boobies
  • Cleavage of boobies (in flight)
  • bra, brassiere of boobies (nesting boobies)
  • nice rack of boobies (perched on mile marker buoy outside of The Dry Tortugas)
  • nice setof boobies (a nice set of boobies)

presentation of bowerbirds

renegade of rule breakers

  • page of birb misidentifiers

mound, infield, sacrifice of Buntings

  • Rib of buntings (perched on a branch)
  • flake of snow buntings (in flight)

Blizzard of buzzards

Deck, conclave, college of cardinals

  • pope of cardinals (a single cardinal, by itself, clearly pontificating to the other cardinals, clearly the leader of the conclave
  • church of cardinals (a group of cardinals, gathered around anther cardinal who is the pope
  • communion of cardinals (more than one cardinal, on a feeder
  • atheism ofcardinals (a group of cardinals at a feeder, with pyrrhuloxia mixed in)
  • pedal of cardinals

clouder, caboodle of catbirds

  • cacaphony of catbirds (perched)
  • catastrophe of catbirds (in flight)

Plateful, winner winner of chickens

  • KFC ofchickens
  • a bucket of chickens
  • throw of chickens (in flight)
  • barbeque of chickens (a cookout in your backyard, where chicken is served)

Syrup of choughs

  • drop of choughs

Bar full of chicks

island of conies

Nursing home of coots

  • convalescence of coots
  • Cuteness of coots ( swimming, diving, sharing a meal together)

glut of cormorants

  • thread of cormorants (in flight, all in a row)

corral, stampede,herd of cowbirds

Bogey, Kellog, truck, case of Crakes

of cranes

  • wrecking of cranes (in flight)
  • ball of cranes (nesting)
  • hoisting of cranes

Clever of crows

ratchet of cuckoos

  • apocalypse of [channel-billed] cuckoos (perched, and presumably keeping David Kirshner awake)
  • delirium of cuckoos (perched)
  • asylum of cuckoos (a wall of clocks)

Salon, iron, perm of curlews

dickhead of dickcissels

dollop, buffet of dove

  • Sound of doves ( when the doves fly )
  • dundering of doves
  • procession of mourning doves
  • funeralof mourning doves
  • dirge of mourning doves

coven of dowitchers

Dazzling, or Parade of ducks

Henley of eagles

Lifetime of egrets

  • egress of egrets (in flight)

emulsion of emus

  • embarrassment of emus

enigma of empids

sallyrand of fantails

Abercrombie of finches

  • frown of finches (hanging upside down
  • smile of finches (perched right side up)
  • pinch of finches (in flight)

flamboyance of flamingos

guillotine of flycatchers (AKA kingbirds)

  • infield of flycatchers (perched)
  • hawking of flycatchers ( in flight, catching insects )
  • fiskar of scissor-tailed flycatchers

wetting of gannets

gorge of geese

  • graffiti of geese (flying)
  • prosper of geese (flying in a ‘V’)
  • ryan of goslings
  • gluttony of greater white-fronted geese ( e.g. “after years of near-misses and as a nemesis bird, we had a gluttony of GWFs” )

comedy of godwits

glimmering of goldfinches

  • gossamer of goldfinches (in flight)
  • midas of goldfinches (perched)

of goslings

Annoyance, snap and pop of grackles

  • cackle of grackles (in flight)
  • armada of boat-tailed grackles

grumble of grouse

Amar Ayyash of gulls

  • frolic of gulls (more than 1000 gulls, in flight)
  • gallows of gulls (floating or sitting on ice)
  • flock of sea gulls

romance of harlequin ducks

Coming, watch, spuh of hawks

  • Smithy of coopers
  • Readiness of Red-tailed
  • Circus of harriers

house, bordello of hens

junky, hunk of herons

brothel of hoary redpolls

hula of hoopoes

Chorus of hummingbirds

  • hymnal of hummingbirds
  • forgot-the-danged-lyrics of hummingbirds (in flight)

Mob of jays

  • blunt of jays (heckling something, harassing another bird or animal or just saying rude things to you)
  • bunt of jays (in flight)
  • sky of blue jays
  • constellation of stellar jays

yard of Juncos

Crown, kingdom of kinglets

castle of kingbirds

Locker of killdeeers

lovefest, embrace of kiskadees

tangle of knots

cackle of kooka

waterbowl of lapwings

lick, lark of larks

  • riddle of larks (on the ground)
  • rodeo of horned larks

bin of loons

bakery of magpies

  • muffin ofmagpies (feasting on a carcass)

malarkey of mallards

submersion of mergansers

shaft of mynas

florence of nightingales

mighty, nutcase of nuthatches

  • crack of nuthatches (in flight)

lasting of orioles

  • nabisco of orioles

government, whom, holler of owls

  • Flurry of snowy owls (less than ten flying)
  • blizzard of snowy owls (more than ten flying
  • snowball of snowy owls (two or more, sleeping)
  • Hootenanny of owls (mixed species)
  • Flurry, blizzard of Snowy owls
  • Conspiracy of burrowing owls
  • keebler of elf owls

parrot of parrots

  • polygamy of parrots (hundreds, feeding in a tree)
  • scoundrel of parakeets

family of partridges

  • pear treeof partridges (perched)

appeasing of peacocks

  • argus of peacocks (waddling around)

platoon of pelicans

  • Formation of pelicans (flying a tight V)
  • squad, attack of pelicans ( large mass of 30 +, coming right at you )
  • mess of pelicans ( eating and dabbling )

tuxedo of penguins

  • slip of penguins (swimming and diving and being penguins)
  • tailor shop of penguins (in a zoo))
  • march of penguins (marching)
  • priesthood of penguins (a large huddle of males, sitting on eggs)
  • pod of penguins (a collection of penguins on an ice float.  AKA an Orca snack)

thundercloud of petrels

  • litre of petrels (swimming)

Fiefdom of pheasants

  • Nook ofpheasants (on the ground)
  • banquet ofpheasants (when flushed)
  • phantasm of pheasants (in flight)

poopage of pigeons

pleading of plovers

  • dojo of mountain plovers

ptooey of ptarmigans

  • silence of ptarmigens (because the p is silent)
  • correlation of ptarmigans (in flight)

cushion of puffins

Stairing, case, bogie, truck of rails

  • flight of rails (in flight)

Nevermore of ravens

  • quoth of ravens

Twist of Red Knots

  • in-flight a darning of Knots
  • on a mudflat: tangle of knots

dire of rheas

wiley of roadrunners

  • cartoon of roadrunners
  • marathon of roadrunners

bat cave of robins

  • nightwing of robins

rockslide of rockhoppers

king of rooks

  • checkmate of rooks

rabble of roosters

pool of secretary birbs

pitcher of shearwaters

snip of snipes

  • sniffle of snipes (in flight)

jack, spartan, chaos of sparrows

  • chaos of sparrows (in flight)
  • muddle of sparrows (muddling about on the ground)
  • plague of grasshopper sparrows

constellation, galaxy of starlings

  • clarice of starlings ( i admit, i don’t understand this one )
  • spittle starlings ( on the ground, puttering around )
  • standing of starlings ( when they are all just standing there, looking at you, terrifying you )

nursery of storks

swig, mouthful of swallows

  • throat-full of swallows

Swindling of swans

  • Fool of Trumpeter Swans
  • casino of Trumpeter swans (swimming)
  • swath of swans (flying)

of swans (flying in a ‘V’)

fleet of swifts

  • laden of swifts (in flight)
  • unladen of swifts (perched)
  • meeting of swifts
  • lost viginity of swifts
  • taylor of swifts

brace of tits

  • brook of wrentits
  • bra-full of titmice
  • handful of tits

squeal of teals

  • sprinkling of cinnamon teal

tuple of terns

  • nacht of black terns
  • tomtom of terns
  • intersection of terns

combine of thrashers

pardon of turkeys

  • Tug of turkeys (immature)
  • Tumbler of (wild) Turkeys
  • Tuttle of turkeys (crossing a road)
  • Cincinnati, splatter, falling, of turkeys (in flight)
  • feast of turkeys (thanksgiving day dinner)

tastiness of turtledoves

a tangle of turkey vultures

a virtue of vireos

  • carillion of Bell’s Vireoe

velocity of vultures

  • mortuary of vultures (on the ground, feeding
  • crematory of vultures (in flight)
  • kettle of vultures (mixed species, in flight, wheeling about)
  • whistling-teapot of vultures (projectile vomiting, in flight, directly overhead.

kennel of wagtails

whinney of Warblers

  • wobbling of warblers (in flight)
  • dècor of Kirtland’s warblers
  • apology of canada warblers
  • thicket of common yellowthroats
  • ikea, roasting of ovenbirds

shedding of waterfowl (on the water, mixed species)

  • pooping of waterfowl (on land)
  • wind of watererfowl (in air)

pigeon of wigeon

happenstance of wildfowl

Weinstein of woodcocks

  • hard-on of woodcocks
  • erection of woodcocks
  • condom  woodcocks

percussion, orgy, of woodpeckers

  • drumming of woodpeckers
  • awkward splinter ofwoodpeckers (nesting)
  • drawing of sapsuckers
  • booger of flickers

Dribble of wood peewees

houseful of wrens

  • papyrus of sedge wrens

Of cranks and origins and things

Questions to ask your kids:

1. What is a “dial tone”?
2. What is a “busy signal”?
3. In cars, why do we say “crank the car window down”? why do we use the word “crank”? maybe you use the word “roll?”
4. What does it mean to “tape” something, as in, a video?
5. When you “turn the heat up” it means to make it hotter, but when we say, “turn the AC down,” what does it mean? (hint: different things to different people)
7. What if we say “crank the AC up?” Does that mean, “make it colder?”
8. Why do we say “turn down” the TV? does anyone have a TV that has anything on it that turns?, it’s regional)
I wonder how many questions my dad could ask me like this, that I wouldn’t know the answer to…

The Explorers

The hazy morning sunlight filtered through the rip in the cheap hotel curtains. It stared at the vanity, reflecting from the mirror to the lamp across the room. The sun was always low in this winter sky and as the beam of light slowly traced a reflective path across the mirror, a misshapen mass of lumps and bumps under the covers rolled over and snored loudly.

Outside, hover-jet taxis whisked past, tearing up the virgin plains, whipping around new construction sites. Workers vibrated their many antennae at the taxis and continued to turn and shape the minerals of the ground in the way the Explorers had taught them. The beam of light gained intensity as the sun rose higher.

Living above ground was more comfortable, more cost efficient, more convenient, taught the Explorers. Better to cultivate, propagate, invigorate the land. “They break our way of life like they break fast: Without a thought,” leading Yibishar complained.

The beam of light reflected across the room to the television (an ancient device, but one which would whet the Yibisharistis appetites) before working its way across the plain white adobe wall, really a textured, virtually unbreakable ceramic derived from minerals present in the indigenous soils.

The phone began to ring and the sunbeam worked its way across the width of the bed. The lump in the bed groaned, rolled over and reached for the phone. He opened up his eyes and groaned as the light hit them. Momentarily blinded, he knocked the phone off its rest and yowled. He looked around. His vision was blurry, but he knew this hotel room. He didn’t quite recall how he’d arrived in it, but he knew it. He shook his head. He had a vague recollection of how he’d ended the previous evening: gambling, drinking, and spending most of his credits. And then the Explorers had come over to his table and started buying him drinks . . . Aury groaned and immediately threw the covers off the bed and checked himself to make sure none of his parts were missing: feet, hands, arms, legs, eyes, ears, teeth, noses, hair, . . .wait! he stopped and began to groan . . . there was something not quite right here . . .eyes?! He’d only had one eye when he’d gone to bed last night!

Quickly he jumped up and ran across the room to the vanity mirror. There it was. Set right there off to one side of his face. His other eye had shifted over too, and his vision—usually double in the morning anyway—blurred in and out of focus.

A knock sounded on the door.

“Room service!”

“Go away! I’m sleeping!” Aury grunted and looked over at the phone by the bed. He looked in the mirror again. I do hope I’m just imagining this. Maybe I’ll just wake up and this will not have happened. His antennae twitched at him. He turned and walked back over to the bed, sat down and picked up the phone off the floor.

He was beginning to get accustomed to the implanted eye but, nonetheless, it took him six tries to get the number right. He held the phone and as he waited for an answer he picked up a glass with one hand, a pitcher with another, turned on the water with a third and scratched his secondary thorax with a fourth.

“You’ve reached the Exploration Command Center phone menu! Please enter the extension number if you know it or hold for more options . . .Press One for Base Operations, press two for Base Operations Orderly room, press three for Base Operations Commander, press four for Base Operations Vice Commander, Press five for . . . . . press seven-hundred fifty seven for Aboriginal Assimilations Research and Alterations Developmental Division — Thank you, your call is very important to us and will be answered in the order received by the next available operator.”

“Dum dum de dum dee ddeee diddle dee dum . . . ” Aury hummed along with the music on the phone. He was familiar with it from his previous contacts with the base.

He leaned back on the bed, setting the phone down next to his ear. On the night stand was an interesting book called Gideon. He picked it up and began to look for good parts. Soon he was asleep, and dreaming.

A long hall stretched before him, and as he walked it turned and twisted this way and that. Each time a turn presented itself, another turn appeared in the other direction, beckoning him. He never took the other turns–he tried to stay to the right, less three hundred and sixty degrees. He knew he wasn’t going in circles. As he passed each turn, billions of tiny men would jump into him, hitting him and pulling at him and he watched how, as he progressed through the maze, parts of his body would be transformed.

“AARADD, can I help you?”

“Sxxxxgjkkkzzzzzzz” Aury snored into the receiver.

“Hello?”

“Snnnnxxxxzggggkkkkllllgrrrgh.”

“Well, ok then. It’s been my pleasure. Click”

“Snxx–Huh? What? Oh crud. They hung up.” Aury looked at his watch and noticed how incredibly late for work he was. Then, as a piece of the previous evening clicked into his jigsaw memory. its puzzled edges clicking firmly into flashback status, he remembered the reason for the previous evening’s binge.

*****

Migration season had passed, and the Zibishari prides had returned to their respective den-hives with much ceremony and celebration.

Aury requested through the retirement formality a leave of absence, was granted one, and settled into his den for deep chrysalis. He would emerge refreshed, having shed his exoskeloten. He emerged bearing an even deeper, more regal, shade of burgundy coloring than he had hoped for, and departed his den to present himself to the queen.

Approaching the hive from a distance, one could almost mistake the forested palisades for natural formations. Upon closer examination, one notices the uniformity of the cliff faces, the regularity of the tiered forestry, and then, as one nears even closer, the teeming activity at all levels at the many entrances to the hive. The first thing Aury noticed as he approached his base Hive was the absence of the usual activity. He checked the sky and noted by the relative position of the moons that he was not early—in fact, he was several days late.

The main entrance to the hive, usually rippling with activity, was nearly deserted. Aury paused at the collonaded entrance and approached a Zibishar who seemed to be standing around doing nothing. He presented himself in the traditional manner, a defensive posture followed by an untenable one. The other Yibishar ignored him and walked away. Aury’s antennae ruffled and twitched. He was amazed. The traditional greeting was never ignored. To do so was to bring tragedy to your entire family for seventy-seven generations. Aury continued on, passing only two other Yibishar as he progressed deeper into the hive. A frightening noise sounded from behind him and quickly he hid behind one of the many pillars lining the Great Hall. An object, whirring and screaming, broke the silence and whizzed past. A strange creature appeared to be trapped within it.

Aury continued on to Central Registration. The area was devoid of life. Aury signed his name in the book, and continued on to his office in the Queen’s Security Council Center only one level removed from the Queen’s chambers. Here, in the Center, he would spend the remainder of the year until the Great Migration would take them to the warmer climates which would induce Yibishar to breed. He was, officially, one of many official consorts to the Queen. In actuality, he was more of a security analyst and spent most of his time purveying security reports.

Footsteps echoed in the halls. Warily, Aury turned quickly; a hideous, pithy looking creature loomed over him. Aury began to ululate the Yibisharisti Fighting Chant and reared up into a defensive posture. His talons wrenched forth from their protected sheathings and lunged toward the foul beast. Aury fully intended to rend the intruder limbless and headless without even saying hello. It was quite frightening looking. Two eyes, two legs, two arms. The simple repetition of the number two so many times was reason enough for Aury to want to destroy it, but then their was that hollow, rancorous noise it was making at him. Like an incentive saying “go aheaad, tear me to pieces!” Aury lept at him. And bounced right off. One of the creatures appendages must have snaked out at him with lightening speed and countered his attack. Aury lept again and again he was repelled. The creature continued to make that grating noise.

“Ok,” the creature said after Aury had bounced off of it several times.

“You speak, beast.”

“Of course I do. I’m a highly civilized creature. May I help you up?” he asked as he extended a hand. Aury rolled up into a ball. When he unfurled an hour later, the creature still stood their. It looked to be on fire.

“You are burning.”

“Nono, I’m smoking a cigar,” it explained and then began making that insidious noise again.

“It smells foul. Why are you making that horrible noise at me?”

“There was nothing else to do while you went fetal on me,” he explained as he stubbed out the cigar underfoot. “That noise is called laughter. It has to do with an emotion called ‘happiness’. But listen, before I scare you again, I was sent down here to gather your security files.”

“Who are you?” asked Aury.

“You really don’t know? Geez. You must be the last person on the planet to be notified. Let me think . . . ” the Explorer tilted his silly looking head to one side and pondered carefully. “Here’s the schtick: I’m an Explorer. I come from a distant galaxy, far far away.” He lit another cigar and tilted part of his head to one side. “We come here in peace, to show the Yibishar a more convenient and self reliant lifestyle—we’re like, I dunno, like gods I guess. But we’re not. We just have a lot of scientific advances that may make us seem godlike. It’s tough, and the Yibishar don’t possess the technology, but I can be attacked and destroyed. Eventually the Yibishar will be technologically progressed and gathered into the Explorer Way, and will join us in searching the Galaxy for intelligent life,” he finished. He took off part of his head, much to Aury’s disconcertment, and waved it toward the ceiling. The Explorer noticed his trepidation and explained. “It’s a hat. A garment, something you wear. See?” He handed him the large brimmed object which Aury then turned over several times in his hands before giving it back.

“Where do I fit in?” asked Aury.

“Ummmm . . . I need those files?” Aury nodded his head succinctly, turned to his office door. and then slid back a series of levers that, when thrown in the proper sequence, unlatched the door.

“There they are.” He directed the Explorer to a row of file tablets against the wall. The Explorer then pointed an object at the files and Aury watched as they disintegrated.

“That was my life’s work! You destroyed it!”

“Nono, just gated it to Howard Space Port. We’ll probably be needing your help over the next few weeks to answer any questions we may have. Here’s my card. You can catch a cab to the Howard Shantytown and stay there until we call for you. If you have any questions, dial the number on the card and connect to Alien Security Control. It’s all on the card in Yibi and in Galactic. Have a nice day!” The creature turned and walked away. Aury stood their looking at the card. He hadn’t comprehended half of what the creature had said to him, but he was beginning to assume that much had happened in the time he had slept.

******

 

Getting dressed was a bit of a chore: his eye glass didn’t fit him anymore and everything looked fatter than it had before. He banged his head on a few things but after he managed to find his way around he began to suspect that the additional eye was indeed a blessing. His peripheral vision improved, depth perception heightened, and he imagined that he would actually be able to see further than before.

After checking out of the hotel, Aury paged a cab and waited patiently until one drifted down from the sky. The door wooshed open. As he reached for the door to pull it closed after him he entertained the sickening observation that as he reached for the door the arm he tried to use to reach for it simply wasn’t there.

“Yiyiyiyiyiyiyiiyiy—” he started to chatter in his native tongue the death chatter he had been taught as a child in hatchling school. His whole body felt like it crawled with insects one instant and numb the next. Other Yibis waiting at the platform looked at him as if he had lost his mind so he stopped, regained some composure and used one of his remaining four arms to pull down the door.

“Problem, buddy?” asked the Yibishar pilot.

“Nono. Just a small matter of growing and losing body parts.”

“Well, as long as it ain’t nothing serious. I knew a guy once who got pinched like a pimple by some heavy machinery. The sad thing was that he was alive, and they knew he would stay alive until they separated the machinery that had him all caught up, ya see? So they done brought his woman by to say goodbye, and his kids and his mawmaw and all. Damn near had a family reunion—”

“Thank-you for sharing, a lovely story, really, but I don’t think I need to hear anymore. Just take me to Howard Space Port please.”

“Ah, you’re a zonie, are ya? Should have known by them two eyes you got. You almost pass for a Zibi, you know. Why, if your shell was just a shade darker…And I can tell them antennae ain’t real . . .if you ask me they ought hire a guy like me to check over what they do over there in alien research and development—You know what them crazy space aliens are doing over there now, don’t you?”

“Please, tell me,” inquired a skeptical Aury.

“Well, I ain’t suppose to tell no one, but seeing as how we’ll all know sooner than later, I guess there be no harm in tellin you,” he confided in a near whisper. “I heard from a couple of explorers that the latest invention the Explorers are going to let us have are robots.

“Robots? What are those?”

“They’re like little, mechanical Yibishar who will do all our work for us. I heard they can even perform surgery, cutting us open and the like to fix what’s gone wrong. I bet they could have saved that pinched guy.”

“Yes. I’m sure they could have.” Little Yibishar . . .little men? Aury felt like chattering again. He began to scratch at his skin. Skin?

“Well, here we are. That’ll be twenty, Galactic.”

“Twenty, Galactic!!!!!?” What do you take me for, a damn zonie? Do I look like some kinda Space Bopper to you?!” he demanded. The Yibishari in the front seat turned around and looked him up and down.

“Well, you sure look like an Explorer, but I don’t know, you speak Yibi better than any Explorer I done heard. A bit stuck up, like you might be one of them Nouveau rich politicians I read about, maybe you been getting some of that surgery or something. That’ll be twenty, Galactic.”

“—oh, nevermind,” Aury handed him the twenty and got out of the cab. He paused as he exited the cab and viewed the Space port sprawled before him. A series of well planned out architectural buildings lined both sides of the street. A gatehouse stood to one side of the avenue and an arch spanned the road bearing the name “Howard SPB” in large, metallic letters. Aury took a step towards the gate and as he fell forward and his face rapidly approached the pavement, he noticed that he was missing a pair of legs and that his second thorax had completely disappeared. In fact, he surmised as he lay there, his exoskeleton was feeling rather mushy and soft as well. Aury looked to the gate and remembered the trees and rolling hills which had once been here. Now they had been hacked back to a safe distance, the jungle kept at bay by stasis field pods set all along the perimeter. In the distance a space shuttle launched vertically into the sky. A twinkle and it was gone. Overhead a flight of hover-craft thundered past.

Aury stood up, brushed himself off, adjusted his now loosely fitting clothing and suddenly wondered at another appendage he seemed to have sprouted in the area where his two walking appendages met his…waist. Oh no, not one of those, too, my oh my. What is happening to me? What have they done?

He took a few hesitant steps and realized that walking wasn’t really so tough. He could even manage a little sway of his arms—he only had two left now—just like the Explorers did. Why am I coming here? he wondered to himself. It’s not like they’ll help me . . . I’m sure this is part of some plan they have to destroy our culture and takeover. They’ll just somehow transform all Zibis into bipods.

He neared the gate. As he approached, a uniformed officer stepped out of the booth. Smiling, the man stepped up to him and extended a hand. “Well, Major O’Donnel,” said the uniform. “I’m glad to see you made it back. We certainly didn’t expect you so soon. Your implants notified us of your approach. I have a car waiting.” He held out an open palm to one side and indicated a waiting car.

“Thank-you,” replied Aury. “It’s been a rough mission. I’ve obtained the information . . . Is this car secure?”

“Of course,” replied the officer. “Central Alien Investigations Office, please. And be snappy,” he directed to the driver. “This man has just come back from intensive transmogrification.” He turned to Aury. “I imagine you are tired, and a little disoriented, and would like to unload the pico-lattices as soon as possible.”

“You bet. Up until I saw you at the gatehouse I had no idea I was really a human, transmogrified into an alien transmogrifying back into a human. I really believed I was a Zippy.”

“Hehehe. How quaint,” remarked the officer politely.

“You know, they actually are amazed by miniature robots? Imagine what they would think of pico-lattice technology. Billions of subatomic energy lattices linked, working together to alter the very pattern that binds us to what we perceive as reality, changing Plankett structure of any object. What next, hehe. Right?”

“Right. What next?” the officer laughed quietly under his gloved hand.

Aury leaned back in the seat, ran his fingers through his hair, and let out a sigh. Wait a minute . . .he ran his fingers through hair again and felt the small, nubby stumps where the antennae had once sprouted. Quaint?

 

Top Ten SQL Server Flameouts

 

This blog post is a precursor to a presentation I’ll be giving at SQL Saturday in NYC on May 30th, 2015.  For those of you who will be attending the conference and this session, be warned that this blog post contains spoilers!

I began working for kCura in 2009 as an Application Support Specialist.  I was hired in a senior role, and over the next few months I worked with 6 other application specialists, back when training was baptism in fire and our clients’ SQL databases were just starting to grow at a very rapid rate.

Our key challenges then were using the right tools, understanding how to read the tools, and working together with our clients.  There were some very long conversations, and some very, very deep dives into performance tuning across all aspects of the platform.  When problems were experienced, if SQL experienced an “event” of some sort, such as a shut down, a stall, blocking, whatever – simply “getting through it” was never enough.  We wanted to know what caused it.  We had a burning desire to find the root cause.

Over the years, a few recurring themes surfaced.  Of course, we had a couple of oddballs – things you may never see or may even believe can’t possibly have happened.  You are entitled to your own opinion, I am not here to argue the points of fact or the historical record.  I am here to tell a story, a story of 10 SQL server flameouts.

Firstly, you may ask, “What’s a flameout?”  Analogous to when a SQL Server “crashes” or loses it’s ability to perform it’s primary function – which is to run and complete queries against a database – a flameout is a non-technical term for the loss of “flame” in a jet-engine.  It can be caused by any number of things – a failed fuel pump, a fire, bird-strike, etc.   This is not a blog post about top ten jet engine flameouts, though.  It’s about SQL.  The bottom line is that if a fighter jet, with just one engine, has a flameout – it’s lost all propulsion. The same could be said for a SQL server.

Fatal Error 211 (flameout 1 of 10)

This is about corruption, and the sudden discovery of it.  This blog post was a direct result of an assertion that was made that the corruption was caused by a log drive becoming full.

This problem did not happen in the logs.  Data on disks becomes corrupt just sitting there.  Data on drives can become corrupt, even when they are not under power.

Log files becoming corrupt isn’t an issue, either.  There is this trick where you can rebuild a missing log, and that is the topic of a later flameout.

There were other questions about what caused this:  Could a corrupt document or load file cause this? In the application that loads data to SQL. data is stringently typed, and there is no character or combination thereof that could have caused this.  Documents also don’t get loaded into Relativity, just the metadata from them, which goes into SQL via typed variables.  As for why the corruption didn’t cause trouble until recently, there was a restart.

The most probable theory of why corruption appears after a restart is that the (clean) system table was cached in RAM and was different than the (corrupted or missing) one on disk. When the server restarted, the instant SQL tried to look at that bad table, it choked.  SQL always looks to RAM for the most recent data, and if a table has cached, it will never go back to disk for it unless it is changed.  Data on disk NEVER changes before it changes in RAM, so SQL would never have any reason to go back and check the table on disk.  In other words, SQL looks for changes in RAM and writes them to disk – not vice versa.  It knows from the SQL log where the most recent data is and whether or not it has been written to disk.  This relationship is at the foundation of something called ACID – atomicity, consistency, independence, and durability of transactions.

This also explains why the backup was bad.  Backups always get the most recent data, and it gets it from disk, and since it had no reason to believe the system allocation pages had changed….

The corruption may also have been noticed (suddenly) if something changed the system allocation pages in memory during the week since the last checkDB.   As soon as it went to commit the change, to that corrupt spot on disk that the DBCC was checking, then the roof caved in.

At this time, there are no known unpatched defects in SQL Server 2008 that cause corruption.  Certainly, a defect in SQL could be at the root cause, but the more likely instigator is the physics of electromagnetism and entropy.

RAM – maxed out (2 of 10)

We’ve all done it – we’ve all been guilty of thinking that some setting or another on our SQL Server is a “set-it-and-forget-it” (SIFI)  setting.  In fact, we have a 60+ page guide dedicated to optimizing SQL for running Relativity.  This guide serves as a good starting point.  Key words are “Starting” and “point.”  Most settings in SQL will need adjustment based on differing workloads – even in the same application. If you don’t know why you are locking pages in memory, and you don’t know why you are raising your minimum memory value, then you shouldn’t do it.  You should get help. Professional help.

A couple of key things to know:

  1. There are three RAM counters that are presented on the performance tab in task manager (Windows Server 2008):
    1. Cached RAM = RAM that was in use but isn’t in use now and is available for use.
    2. Available RAM is roughly = cached + free
    3. Free RAM
  2. Total RAM = A + B + C
  3. in Windows Server 2012:
    1. In Use RAM
    2. Available RAM
    3. Committed RAM
    4. Cached RAM

You shouldn’t necessarily be overly concerned about Free RAM hovering on or around zero, unless of course this is a new thing that is not normal for your system. When Available RAM continuously hovers between 0 and 100 MB, you have a problem that you need to fix.  Unless, of course, this is normal for your system and you are so awesome you can walk right up to that line and flirt with it as though she were superman’s wife and you have a bucket of kryptonite in your left hand and a crowbar in your right.

Without exception, when I have seen Available RAM at 0 and had a sluggish and non-responsive system on my hands, this was the problem.  Most of the time, my first action is to lower max memory in SQL first, and ask questions later.  Why?  Because I need a system that is responding to my actions in order to figure out what happened.  SQL server is usually very responsive to this action, and the system becomes almost immediately useable. Then, with the client and other support representatives from kCura, we would go on our scavenger hunt and figure out what had happened.  It can be anything from a bloated file-cache – a user dragged and dropped a large file onto the desktop of the SQL server, which over time erodes away at the RAM for the SQL Server, or maybe they had web browsers open and were generally treating the SQL server as though it were a desktop machine.

By the way, its a good security AND performance practice to prevent Internet access from your SQL server.  Furthermore, you should not have SSMS even installed.  If you can run Core,so much the better and congratulations to you for this – it is a great practice.

Conclusively – keeping an eye on your max memory setting, and knowing who and what are stealing precious RAM from SQL, is a top ten skill in the DBA toolkit. Knowing whether or not you should even set Min memory and if you should use lock pages in memory – that is an advanced topic and I have my own opinions about it, some of which are unpopular and if you can find me after my SQL Saturday session, I will happily discuss it with you.

Stuck in Recovery (3 of 10) 

Hands-down, this is probably one of my all-time favorites. All Relativity databases (with a couple of exceptions) ship, by default, in FULL recovery model. Sometimes it becomes necessary to restart SQL (I know, right?) and if we don’t have the healthiest of log files, restarting can become a most painful process.  You can read about log file maintenance here (link to relativity tech support document about Relativity log file maintenance) which is a document we created and has been peer reviewed by Brent Ozar and Mike Walsh.

If you are not doing anything special to maintain your log files, you are missing out on some of the sweetest performance benefits you can have.  Aside from decreasing restart times, and not having your database get stuck in recovery for 2 days (yes, this can happen), you can boost performance by removing LOG WRITE WAITS – completely.  Yes.  You can COMPLETELY remove LOG WRITE WAITS.  This is not a WAIT type that should ever be in even your top 100 WAITS, unless the WAITS were created by some maintenance that you did. (you of course can’t COMPLETELY remove them.)

This link takes you to a SQL script that is useful in counting how many VLFs you have.

And here is a post by Jonathan Kehayias  that explains log management more thoroughly.  If you plan to grow your log file organically – that is, just set it at some size and then forget about it, and if you expect the log file to grow, then you should consider to set the growth size to be big, such as 512 MB or 1 GB. Consider that, if this becomes a problem and you begin to see LOG WRITE WAITS, then you should make the decision to attempt to predict how big your largest read/write table with a non-sequential primary clustered index will become, and you should force-grow your log file somewhat larger than that.

“Why Read/Write, non-sequential?” you ask?  While it is true that there are a few outlier situations where a linear key, clustered index on a large table may need to be rebuilt, mainly the log file space operations that will consume the most space are as follows (not in any necessary order):

  1. non-sequential, clustered index rebuild – it’s so fragmented that only a complete rebuild of the index will do any good.
  2. non-clustered index re-orbs and rebuilds

There are probably other things that will chew up a lot of log space, but as a good DBA you should know what they are and be prepared for when they will happen.  You must be able to anticipate these things. The main reason for concern in Relativity about the type of table that may be rebuilt lies in our auditing structure.  Every single user action, every data load, every mass edit, is logged in our audit record table.  While you may  (in Relativity) have a 300 GB Document table, you may also have a 3 terabyte auditRecord_PrimaryPartition table.  This Document table, which is read/write, may have some very large non-clustered indexes that need to be rebuilt, and has had on occasion needed to be rebuilt due to the addition of computed column indexes.  The auditing table, conversely, has realtivitley small, non-clustered indexes, and its primary key is monotonically increasing and is of little concern.  This table will never need to be rebuilt, and if it does, then you will know about it.  It is not something that will happen in the middle of the day or over a weekend in a maintenance window.  If the non-clusred index on this table has actually exceeded the size of the Document table, then that would be an interesting thing, and we would like to see this and of course you would then adjust your behavior accordingly.

So, these are just the lessons we have learned. and they may or may not be applicable to all databases everywhere.  SIFI is not in the cards for any of us in this room, we must learn from experience and make wise decisions based on past performance.  This is the best we can do, and while I can share with you my experience, and the experiences of kCura, I can not give you SIFI guidelines.  It is the sign of an immature DBA that would even ask for such a thing.

“hey!…that’s my turbo button!?” (4 of 10)

If you call into your software support representative, and you have turned on Priority Boost on your SQL server, you may expect to be asked, quite firmly, to remove that setting.  If you refuse, you may find that your tech support company may refuse to offer you any further assistance regarding your issue until such time as this box has been unchecked and SQL has been restarted. This feature marks SQL Server threads with the highest priority.  No other processes, including those of Windows, will get a higher priority.  Combine this with someone setting MAXDOP to 40 on a 40 core server, and you have just ghosted your SQL Server. Flame.  Out.

Paging (5 of 10)

This relates somewhat to flameout 2 of this series. Sometimes, if SQL comes under memory pressure, it will choose to page memory out to disk.  Once this memory is paged out to disk, we have seen that it does not return to VRAM, even after when whatever circumstances that caused the paging have been alleviated. If it is an option, I recommend you restart SQL.  Of course you should know your VLF situation before you do that, because if you restart a SQL server and get stuck in pending RECOVERY for 2 days, you will have a very upset stakeholder on your hands.  This is one of two circumstances where I will recommend a SQL server Restart. The other one is if a SAN gets unplugged in the night.  SQL will show the database as being there, but it won’t be.

Another way to get the cache to clear out might be to runn

DBCC FREEPROCCACHE

DBCC DROPCLEANBUFFERS

but I have found that these two commands do not reliably dump ALL buffers.  You may still have something left in the paged buffer pool that will create significant impedance.  There are probably some additional commands I could enter here.

Cache out (unsolved mysteries) (6 of 10)

ON three separate occurrences, so yes this is a more esoteric top ten item, we have come across a situation where the oldest item in the cache was about 10 minutes old, rolling forward. This means that as we sat and watched, even after 30 minutes of observation, we saw no significant aging.  All the settings were to best practices – plus there is no one setting that could cause ALL items in the cache to never age.

Here is where a DBA that faithfully follows a strict protocol of change control must be lauded.  Were it not for the DBA, able to tell us the ONLY change that had been made in recent memory – we would still be scratching our heads. I had been on site with this client just three weeks prior to this follow-up visit, and the problem of the vanishing cache did not exist then.

He had set MIN Memory = MAX Memory.  We all doubted that this could be the root cause (we still do) but when he set Min Memory to 0, the problem stopped.  To this day, when we talk about it, we refuse to believe that this setting CA– USED this, but since then I have seen it on two other servers. There may be some other environmental factor at play here, something to do with clustering or maybe to do with another configuration setting, we don’t know.  For now, if you see this, change the min memory, which you probably should not have set at all on a stand-alone instance of SQL where it is the only application.  even were this not the case, there is no reason to set them exactly the same, there is no detrimental effect to leaving them a few MB or GB apart.

 

Wait, did you just say NTFS? (7 of 10)

One time I received a ticket that involved optimization. Our client complaint was slow queries.  She was right, the query was slow, and it was a query that I already knew about, and I knew what index to apply to get it to run very fast, sub 100ms.  After optimization, though, the query would not go faster than 1800ms.  “This is still too slow”.  I sniffed around the system, ran some checks looked at WAITS, and it become pretty apparent to me that we had a problem with storage.  I let her know, and a week later we were on a call with one of kCura’s storage and virtualization experts, the client, and me, the tuner.  I began asking some questions and as the client’s storage expert answered my usual questions, a text message popped up.  Our storage expert said “He just said NFS.”

“What??” I replied via text. “I thought he said NTFS.  Who puts their SQL on NFS?”

Next, I asked, “Did you just say ‘NTFS’?” and the client replied, “No, I said ‘NFS’”

“Are you running SMB 3.0?”

“No.  Is this a problem?” he asked.

“Yes, it’s safe to say that we do not need to have any further discussions.  This is our root cause.”

Several weeks later, the client moved on to an SQL server that has dedicated storage.

 

It worked fine yesterday 8 of 10 (top 10)

Inevitably, you get the call, “This query worked fine yesterday, today it is not working at all and won’t finish.”  Here are ten possible reasons.  Can you think of anymore/

  1. The root query fundamentally changed
  2. An index was dropped
  3. The size of the data being queried increased 50x overnight
  4. Storage was swapped out
  5. A setting was changed (Max memory set wrong, MAXDOP set to 40/40 cores, etc.)
  6. The user is confused – their query is brand new and has not been run before today (plan cache)
  7. The server was restarted, the cache is not warm
  8. There was a failover (we’re not home alone anymore)
  9. Maintenance plans have been failing. FOR WEEKS.
  10. We are fresh out of magic pixie dust

Blue Screen (9 of 10)

I have very vivid memories of this issue. I started at kCura in 2009 – up until this point, i had been a paid consultant, where the affected users I served were either employees of the billed party or were a partner in a law firm that would be paying my bill.  Hands down – across the board, they would rather buy a new server than have me spend 20 – 30 hours (or more) digging into a root cause.  kCura has a different approach – we don’t bill our clients by the hour to help them solve their technical challenges.  We just help, and we keep helping until we either have an answer, or everyone agrees that an answer is not possible.

We received a call one afternoon of a server that had become unresponsive.  During the course of the call, the SQL server blue screened.  What followed was a very demanding client, insisting that we explain how and why this happened – they wanted to know what caused it.  Review of the error log showed that SQL server in fact had shut down and we had a windows dump file that we were able to send to Microsoft for analysis.  In the error log, we had a portion of a query.

Relativity allows users to create and run custom search queries against sometimes large data sets.  You are probably all aware of Google’s limitation of some 4o or so words in a search. At the time Relativity had no such limit – and the search query that SQL had dropped into the error log was very large. So large, in fact, that it had been truncated when it was inserted into the error.

This query became central to our investigation.  Two days later, I had completed the design and creation of a query that would search through all workspaces and pull back any saved search that had a size greater than 10,000 characters.  It returned a large number of searches.  I increased to 20k.  Then 100k.  Then 400k.  Two searches came back – and one of them was an exact match of our suspect.  With this information, we were then able to figure out who had run the query, and when it had been run.  We were able to train  the user, and we also modified our application so as to allow a configurable upper limit on search submissions.

By listening to our clients, and by digging deep, we were able to make our application better, and improve the customer experience.  This query I built to analyze searches across all workspaces was the beginning of a view and recurring saved search analyses framework, and its development would span 5 years and result in the complete automation of search analysis.  This one thing that would normally have been swept under the rug in most corporate cultures became instrumental to development that would take place 5 years later.

 

 

Warning: Disk IO took longer than 15 seconds (10 of 10)

One thing is certain – where there is smoke, there is fire.  This error message in your log reinforces the lesson “Check the error log first.”  It is your first  go-to when troubleshooting a SQL problem. Much of what can go wrong in SQL will be reported here, and if SQL Server can’t talk to it’s files for more than 15 seconds, it logs it in the error log.  We had the chance to dive deep into this error on a large, VCE converged system that had been experiencing extensive issues.  Ultimately, we felt the fault lay in the fast cache, that it was stalling and not responding.  During troubleshooting, the client suggested that the disk IO errors were not related, and that we should seek the root cause elsewhere.  This prompted us, in tandem with the client’s assistance, to develop a script that could detect IO taking longer than 1 second.  This involved a little custom scripting and analysis of SQL’s ring buffer.

After the script was completed, and we ran it on a 2 second wait interval, we learned that not only were there an occasional I/O wait that took longer than 15 seconds, but there were 1 second long burps happening almost constantly.

This script that ran this check was not difficult to write, but it has fallen to the doom of too many tabs opened in Management Studio and it wasn’t saved.  According to Mike Walsh (Linchpin DBA) this information can also be teased from extended events.

 

Punch-cards like birding

May 4, 2015

Today, we had Rose Breasted Grosbeaks at our feeders.  This was super emotional for me.  Why?

In the spring of 2009, or maybe 2010, Ethan’s interest in birds took an aggressive twist. Until then, I somewhat thought it a quirk of his childhood. Something he would outgrow – a phase, or a passing interest. One day, when I came home, he was going on about having seen a Rose-breasted Grosbeak at the feeder. He described it to me, and it was a bird I had never seen.

Rose -breasted Grosbeak

I wrote it off as the ramblings of a 7 year old child with an active imagination.  After all, prior to us having bought him an actual bird book published after 1940, (about thirty days before this), everything was a Chipping Sparrow. Besides, I was pretty certain at the time that I would only see chickadees, cardinals, and robins in my yard along with the yucky black birds that make a lot of noise.

The very next day, sitting in my family room, a bizarre looking bird landed on the feeder. I grabbed my camera (5D with a 200mm lens) and took some very grainy pictures through the window.

Interestingly, I had just finished reading Outliers – I’d learned how Bill Gates became a great programmer because he was able to compile and debug code on the fly (as opposed to the tedium of punch cards where a single mistake can take weeks to debug).  I had an idea – I would photograph all the crazy things that Ethan sees, and give him the same kind of debugging feedback. The old way of birding, where the birder sees the bird, then either right there of later on starts flipping through pages in a book, and trying to work from a faulty memory, reminded me of how programmers used to have to work.  They would make a bunch pf punch cards, put them in a stack (don’t drop them!), feed them through a computer, wait for it to compile, and then debug.  With the advent of on the fly compiling, you could program and compile much more quickly. I would do this for my son, he would see the bird (write the code), identify it (compile it), and then check it against the photo (debug).

I became hooked. I already loved photography,  I am fortunate to work for a great, successful income that allows me to travel and purchase equipment, Ethan loved birds, and so I became a review tool he used for verification (birds in flight, distance shots, etc), and I became his student. Through him, I’ve learned to become a better parent and a better photographer. I’ve learned that when you, the parent, fuel the fire that is in your child, when you fan the flames, they become a conflagration, a burning inferno of passion. All you have to do is create opportunity, and get them whatever they need to learn, and.then, the child becomes amazing. If you can do what they love with them, all the better!

Today, after a 6 year absence, the Grosbeak came back (Ok, maybe not the same one) and it brought three friends. We had FOUR Rose-breasted Grosbeaks on our feeder today. Yesterday, the find of the day was in Urbana at Busey woods when Ethan located a rare White-eyed Vireo who just happened to very much want his photo taken.  The RBG and WEV are in this set on Flickr: https://www.flickr.com/…/44211019@N…/sets/72157651937850019/

Personally, I am thinking that the White-eyed Vireo came to see us, because he hung around us and followed us for about 100 feet before flitting away up into the treetops. Maybe he’s friends with the GWW – I’m telling you, they talk about us behind our backs …. 😉
Scott Ellis (Ethan’s proud and slightly insane Dad and personal photographer)

What is eeCardinal?

eeCardinal is the beta version of a photography app that integrates with flickr. It’s purpose is to enable people to review photographs in an intuitive and user friendly manner.  No further information about this app is available at this time.  It is described in some great detail in the book

Mobile Devices: Tools and Technologies – CRC Press Book

If you would like to know more, you must purchase the book and return here.  Further information will not be available until the book is released. If you are a test flight user, you can reach me via email with any questions.

eeCardinal is built on Glimpse360 technology, which is also produced by Scorellis.  eeCardinal may not be the final name of the product.

eeCardinal test flight support

For support on eeCardinal, the namespace on flickr, or on Glimpse360 vector technology, contact scorellis through the self-same address at gmail.com.

Solicitations or inquiries regarding the app development, jobs, photography, etc, are not being accepted at this time.

 

Thurple Thursday Morning

This morning I awakened and as I got dressed, I thought to myself,”Iit’s not waffle Wednesday, why am I putting on waffle? Too bad there’s not a material called ‘thurple’.

“Hey, that would be a good name for my app. ” Then I googled it and found very few results. This one stood out, because my app is about creating an identification space.

http://www.serc.si.edu/education/programs/java/activities/purple.aspx.

Then I said the word out loud. Then I decided it didn’t make sense. The search for a catch name continues. For now it is still eeCardinal.