Yesterday I wrote about the apparent tendency of Texas power plants to literally catch on fire and explode if demand for electricity gets too high. My deeply considered response was basically "Huh?" Today, though, a reader emails to explain things to me.
The story starts with the fact that our electric system is based on 60 Hz AC. This means that the direction of the current reverses 60 times per second, and obviously the entire grid needs to be in sync:
The fundamental reason why grid operators have to resort to induced blackouts to protect their generation stations (and why it's sometimes done automatically) is that AC frequency management is everything in grid safety. Every generation hooked to the grid that uses physical motion (typically a turbine) has huge, expensive, precise turbines turning at some local multiple of 60Hz, and all of them are putting out AC whose waves are synchronized with one another. A small gas plant in El Paso phase peaks within a few milliseconds of when the steam turbines at the Comanche Peak nuclear plants do.
Under normal conditions that's just how AC power works — if it wasn't in sync no one would get power anywhere. Under strain, such as when demand surpasses capacity, frequency starts to drop — turbines start getting more magnetic resistance from their coils, solar inverters start heating up, wind turbines get more rotational resistance from their generators, etc. Most of that equipment can't tolerate going out of phase or frequency target more than slightly before it's damaged in big expensive ways. Hence they'll trip off the grid to protect themselves if needed, and grid operators have to drop loads rather than letting the frequency drop when they're out of generation capacity. They wouldn't "blow up" if they just stayed connected, but the damage would be real. There's nothing special about Texas' grid in that respect — all AC grids work that way.
There's a whole side market in energy just around frequency management — batteries and generators that can add or take away energy just to maintain stable frequency in the phase of short-term load shifts. It's fascinating stuff if you're into that sort of thing.
Another reader writes:
Remember the testing done around Y2K or thereabouts of whether you could actually damage a generator through the Internet? They destroyed a really big diesel generator with astonishing ease by oscillating the load to form a standing wave and ruined it beyond repair. Mind you, diesels are not known for being fragile, and certainly not as fragile as a turbine. These are big, macho machines, and each one is essentially a one-off custom build, so when one is severely damaged, it might take years to replace.
This YouTube explains in graphical terms:
Thanks for passing along this information. It does make the problem clearer.
This video is much better at explaining how these systems work:
https://www.youtube.com/watch?v=RXJKdh1KZ0w
I don't get the semi-boloid slots, though.
Ok--that makes sense for the need for rolling blackouts. As far as I know, plants didn't trip off due to too much demand. However they lost capacity as plants "froze up" or lost natural gas supplies. In that case, there is an emergency--immediate blackouts else plants would trip off. Emergency orders also meant there may not have been time to organize rolling blackouts, combined with the fact that more power generation was being lost even as blackouts lessened demand. Something about compressors needed to move natural gas losing power.
https://www.salon.com/2021/02/19/what-caused-the-texas-disaster-decades-of-republican-deregulation-laissez-faire-run-amok/
My co-workers at an electric company explained to me that many types of power plants have large, heavy parts balanced to rotate at 3600 rpm (or some multiple) with minimal vibration. When electricity demand begins to exceed electricity supply, the frequency on the AC grid begins to drop below 60 Hz, and those parts begin to rotate at lower speeds, for which they have not been balanced. Then those parts can resonate and shake violently, like an unbalanced wheel on a moving car. So far, so clear: The part I don't understand is that the large, heavy parts in power plants have many "degrees of freedom", which makes it impossible to balance them for more than a few rotation speeds.
(Side note: There is a good analogy between transfer of power on an AC grid and transfer of power through the chain on a tandem bicycle. The analogy is presented starting on page 2 at this link: http://www.gonder.org.tr/wp-content/uploads/2015/04/ElectricityTandem.pdf )
My ex was a system operator for PGE in CA. If those huge turbines lose their rotational balance they can literally tear two foot bolts that hold them to to floor off. I've seen photos of a big turbine that did just that. They have to be taken off line or the substation with the turbines will be destroyed.
The shafts of large steam turbines have a resonance mode in bending where they will absorb energy and ring like tuning fork. Balancing helps, but there is no such thing as perfect balance. In practice the turbines are brought smoothly up to an operating speed above the saft resonance, moving through the resonance before the shaft can absorb enough energy to damage itself.
The US and Israeli intelligence services took advantage of a similar situation to attack the Iranian ultracentrifuges that where enriching uranium. Normally the centrifuge controllers slowly vary the rotational speed to avoid exciting rosonances. The Stuxnet virus took over the controllers and would intentionally dwell at operating speeds where resonances were likely, all while putting out false speed readings so the human operators wouldn't get suspicious.
Appreciate this, but I do think that when we're looking at Texas in particular, it's important to stress that the key trigger behind this protective shutdown is the imbalance between demand and ability to generate.
As Kevin's source says, "under strain, such as when demand surpasses capacity, frequency starts to drop." MarkM48 says much the same thing-- "when electricity demand begins to exceed electricity supply, the frequency on the AC grid begins to drop." The destructive rotational forces that can tear generating equipment apart-- the dramatic scene Kevin is focused on-- are set off by the imbalance between power demanded and capacity to supply power.
Other parts of an electrical system may be more affected by current flows, which responders in other threads have said can, for example, cause transformers to overheat and explode. Or cause transmission lines to overheat and sag into foliage and fail, as happened in the 2003 northeastern outage when that failure overloaded other components and led to a cascade of failures. These too are set off by demand/capacity imbalances.
About the degrees of freedom issue, that car tire and wheel will have masses on the outside edge (the one facing out from the car centerline), masses on the inside edge, and masses all around the circumference at positions between the edges and at different distances from the axle. That can be complicated to balance in all planes, but even a small-diameter wheelset only needs to be balanced up to 1000 rpm or so (900 revs a mile @ 60 mph is 900 rpm, and small-diameter passenger tires turn about 850 revs/mile or less).
Then imagine a tall stack of much larger-diameter and massively heavier sets like that turning more than 3-1/2 times faster. There are masses all over the place and in relation to each other at all kinds of angles across the rotational axis and in no relation to it, and I'd have to think that correcting any one imbalance would have the potential to affect a whole bunch of those relations.
To some extent generators tend to synchronize themselves; one that is lagging slightly behind in phase will (relatively speaking) absorb some power to speed up until it matches the others. Interconnected generators not only turn at the same rate, but once synchronized the rotating parts are in damn near the same positions at a given instant of time. But this sort of self-synchronization is only accomplished when they are already very close in rotational position, energy input and output are balanced, and so forth.
For a given rotation speed the amount of energy generated (and thus the mechanical energy required) is controlled by the strength of the generator's magnetic field via adjustments to the field current. And mechanical energy can be controlled by turbine steam pressure, etc. These both give the means to keep the generators turning at a constant rate as load varies. Both on startup and during operation, complex control systems bring the generators into synchronization with each other, and brings them into phase with the power grid at which point they can be connected. However, even a relatively small change in synchronization with the power grid can cause an automatic disconnect, since out-of-phase current can increase catastrophically, which can not only damage the generating station but overload transmission components as well. This disconnection can lead to a domino effect if the remaining power generators can't be kept synchronized. To prevent this, the grid has to turn off part of its load (i.e. blackouts) or bring more power on line. And, of course, if this isn't done Bad Things Happen. Bringing in external power that is not precisely synchronized will only make things worse, which is why Texas' isolated grid was left to fall over on its own.
Thanks for the tutorial. It's interesting stuff.
A few decades ago I worked on user interfaces in control rooms of electric generating plants. The 'instrument' for synchronizing units to the grid was in some cases a couple of ordinary light bulbs. Small transformers stepped the voltage down to 120VAC; one terminal from a bulb connected to the transformer driven by the unit, the other to the transformer on the corresponding phase of the grid. As operators brought the generator up to speed and matched the grid for frequency and phase, the bulbs would dim and then extinguish. Simple and effective.
Back in the early 70s I was studying electric power systems in college (MIT). We were building generation systems, distribution networks, and simulated loads in miniature that operated in "real world" time scale. The materials physics behind that took a lot of work from a number of materials science labs in MIT - normally when you miniaturize things the timing of events goes "off" in various ways.
Creating electronics that could be dialed to simulate steam driven turbines with a number of different energy inputs - nuclear, coal, NG, hydro, diesel, etc. was a major thesis project in its own right. My major project was building an exact scale model of a specific generator at American Electric Power in the US Southeast. With the simulated "turbine" set to coal and steam it took several hours to crank up from "cold start", through pre-warming the turbine, to the point where I could synch to the (Cambridge) grid - using three lights across the 3 phases my generator put out. It could require a good hour or two of jiggering with the "steam" to get everything properly aligned and spinning at exactly the right speed to get the lights completely dark before closing the relays to lock the generator into the grid. (There was a way to bypass the early steps and more directly control the DC motor being controlled by the electronics - but the professor running the lab was a stickler for verisimilitude.)
Or I found that since the generator was so small it was incredibly resilient compared to the truly titanic generator and full size steam turbine/coal boiler it emulated - so I could just snap the relays with the lights flaring and flickering and it would ring like a very large bell and immediately be synched. I demo'd that to a visiting delegation from AEP - looked up and everyone had gone white as a sheet. After a moment of silence the chairman of the board sidled up to me and said: "Son, can ah try that?" About 10 minutes of hootin', hollerin', and bongs followed, because everyone on the board wanted to taste the forbidden fruit.
Doing that with a full size generator is likely (at the least) to wind up shearing some massive blades off the first stage section of the steam turbine and through anything in the way for a few miles. The turbine casing, the plant walls, a train, etc. That actually happened in Japan a while before I got to MIT - something went wrong with the relay closing and it went on the grid before synchronization had been achieved.
Oops, should have been last stage section of the steam turbine. They widen as they get further away from the steam input, and at the last stage the blades are huge and running in what's basically a rain storm. The idea is to extract every possible bit of energy from the steam to convert into mechanical energy for the generator.
The amount of current generated is directly proportional to the number of magnetic flux lines cut per second. So there are three parameters you can vary to change the amount: The strength of magnet, the number of conductive windings on the rotor, and the speed of the rotor. One of these is more amenable to change than the other two. Freshman physics -- it should be a base requirement for everyone 🙂
So Edison was right all along?!
TOPSY DID NOTHING WRONG.
These technical dissertations are truly fascinating, and I see now that the Texas Disaster could happen anywhere. But all these explanations fail to answer the question of why did the designers, system operators and regulators let this happen? This isn't the first time their grid has been compromised due to cold weather, and scientists have been warning about impending extreme events for decades. Why were the pipelines and pumps not insulated nor generation facilities winterized? In short, why did they gamble with something so critical to the lives of their customers? Is it just that reckless screwing around is in their DNA dating back to oil wildcatters?
"Is it just that reckless screwing around is in their DNA"?
Well, allergy to spending money on stuff that would help secure the public's interest, instead of lining their own pockets, might be more like it, but I'm sure that's just me.
It's looking like we'll need to fasten seatbelts for epic finger-pointing and buck-passing now. On a different thread here BobPM said when Perry was at FERC he killed any chance of a national grid system that could have helped handle this kind of situation, and that Abbott had killed implementation of the FERC recommendations following the 2011 freeze. But now Abbott seems ready enough to blame ERCOT for not doing that very thing. Can anyone imagine ERCOT people sitting still for that? Or Perry not going full jackass and figuring out a way to blame federal regulation somehow? And both Perry and Abbott to blame power customers? It would all be pretty entertaining if the backdrop wasn't such a massive tragedy.
Seriously, everything I've seen says the people who made the decisions wanted a system that was all about short-term incentives and maximal short-term returns and no outside interference. They might even have cut out the kind of speculation that Econ 101 says is supposed to substitute for prudence, I don't know.
There are a whole lot of really smart and capable people in Texas, but there are also a lot of short-sighted and irresponsible ideologues. I found the same thing in Louisiana. In both places the ideologues get the big-money sponsorships and the key public positions and leave the capable people to clean up after them as best they can. Public responsibility is an accident, not a thing.
"Abbott had killed implementation of the FERC recommendations following the 2011 freeze."
Well, wait a minute. According to wikipedia, Abbott didn't become Gov until 2015. Prior to that, he was AG. Did the FERC recommendations take that long to come out?
It's just Texas's bad luck that this man-made, predictable disaster came right after an election. By the time the next one comes around, in two years, nobody will remember the Bad Winter of '21.
FERC put out its report in August 2011. It's very comprehensive, about 350 pages (including some really good appendices on topics talked about on this thread), and makes a lot of recommendations.
I can only speculate about Abbott on the basis of what BobPM said, but the recommendations involved changes in how ERCOT and individual suppliers should operate and communicate, and considering some changes to state laws to get some of this to happen. None of the recommendations, as far as I can see, would have changed the basic structure of either ERCOT or the natural gas business; it was all about trying to prevent deep-freeze and other weather problems.
Abbott, as AG, would have had a lot to say about that, especially anything to do with new legislation. At the time, per wiki, he was busy suing the Obama administration about everything under the sun, essentially-- anything that could be set up as part of the culture war. It isn't hard to see how he could have fought against the big bad feds trying to big-foot Texas as if they knew better than Texans themselves how to deal with local conditions, etc.
Even though (as far as I can see) nobody said a word about Texas not continuing to run its own grid, just to try to institute best practices as known from other regions, this kind of reaction would have been perfectly on-brand and salable to the public. And, by purest coincidence, it would save the utilities a bundle and a half that could go to officers, shareholders, fuel suppliers, and political contributions.
Texas buys power from the lowest bidder. Anyone who spends extra money winterizing isn't going to be the lowest bidder, and likely won't be in business for the next bidding cycle.
Yep. And that's why things like weather-readiness tend to be requirements of doing business in other places, overseen by entities similar to ERCOT (the R *is* for reliability), by regulatory agencies, by the state, or by all three. It's like the minimum wage-- if everybody has to pay it, nobody who doesn't cheat gets a competitive advantage.
If a change in rules like this would disadvantage small providers, they can either be given aid toward the costs, or be governed by rules that are sensitive to size and financial capacity. If there are ways to balance opportunity for the little guy with overall system reliability for the public, I'm confident there are people in Texas smart enough to figure them out, once the state commits to the idea of a system that operates reliably in foreseeable conditions.
Syncing gens reminded me of a story a co-worker told be back in the 1960's. He was manually syncing two gens at the LeTourneau earth mover factory. As he was in process another worker told him to shut it down. He asked why? Time for prayer!!!
Also the story about gens running above the first harmonic reminded me of when we purchased a 50,000 cfm 150 psig air compressor from Mitsubishi. They were proposing a German MAN design which has a bull gear with small compressors driven off the gear. We were wondering why they used 5 small compressors. It finally hit me that our spec said the unit had to run below the 1st harmonic. We quickly changed the spec to allow running speed above the 1st. As a side note the Japanese matched our requirements to the letter. Americans proposed compressors of vintage 1930's era. Their documentation was just a jumble of cut sheets. Sad.
For the lucky few who did not lose power, the unregulated utilities of Texas had a surprise of a different kind:
'“My savings is gone,” said Scott Willoughby, a 63-year-old Army veteran who lives on Social Security payments in a Dallas suburb. He said he had nearly emptied his savings account so that he would be able to pay the $16,752 electric bill charged to his credit card — 70 times what he usually pays for all of his utilities combined. “There’s nothing I can do about it, but it’s broken me.”'
https://www.nytimes.com/2021/02/20/us/texas-storm-electric-bills.html?action=click&module=Spotlight&pgtype=Homepage
Why aren't people just refusing to pay? Why isn't there a rallying of gouged consumers who all refuse to pay? Why haven't we heard of any class action -- or even individual -- lawsuits?
Are Texans just a bunch of sheep, ignorant serfs accepting as fate everything governmental incompetence throws at them?
It's a new issue and has not gained momentum yet. I live just north of Dallas, and I haven't seen my electric bill yet. My power went out for long stretches, but when on it may have been some pricey electricity. If many people have really high bills, you can bet there will be collective action to fight back.
Politicians here know people are pissed. Abbott on Saturday held a meeting with other state leaders on this and he said relief is coming. We'll see.
Yeah - relief is coming. They want the rest of the US to pay the bills to/for the TX power companies - anything to maintain the all important flow of money to the coal and NG companies.
Power sources should not have had to be shut down deliberately. The technical reasons why they would have to shut down or why they supposedly came close to disaster are really irrelevant. These reasons are well known, and the action that had to be taken is also apparently well known - rolling blackouts, or partial shutdown of electric demand (not supply). If there were deliberate shutdowns of supply (is this known to have happened?) then the demand withdrawal (rolling blackouts) was presumably not handled correctly. Why this happened is not really addressed in this thread.
This is different from the matter of why some sources were involuntarily unable to supply power. That is now well known - basically lack of sufficient winterization.
Eventually IEEE Spectrum will publish a readable, clear account and analysis of the event.
I stand by my Gell-Mann amnesia comment.
"literally catch on fire and explode if demand for electricity gets too high"
I am both astonished that so many people are unaware of the necessity of perfect AC sync, and that there are multiple layers of redundancy upon redundancy to ensure that that sync is maintained -- no part of which involves power plants catching on fire and exploding.
Once again I would score this as an example where journalism could have provided genuine understanding and value-add, but instead went for sensationalism and collectively making people stupider.
As someone else noted above, “So Edison was right all along?” My engineering expertise ends at the point of noting that Edison advocated for DC current to be the standard mode for power utilities (and Tesla pushed for AC, so Elon Musk is obviously to blame for all of this mess)(j/k Elon). Presumably DC generators would have not needed to worry about any of this frequency resonation concern, since their output is always going in one direction. (There were still a few remaining power systems in older cities that supplied industry even in the 21st Century for the most part that are needed to operate DC only equipment—See https://www.trainorders.com/discussion/read.php?11,1541316— but those are probably all gone now.)
So engineers, school us. Would the lack of need to frequency match all of these generators have been worth it? Or what else says AC is still the wiser choice?
Edison couldn't produce high enough DC voltage to efficiently send power a long way. At the end of the transmission line, at the load, the DC voltage had dropped substantially. So rather than the power being used by the load, it was dissipated as heat in the transmission lines and wasted. To transmit power with low losses down transmission lines, high voltage is needed.
Electrical power dissipated in power lines depends on both the current in the lines and the electrical resistance of the lines. If very high currents are in the lines, the electrons flowing (the current) will have more obstacles to overcome (the resistance). This produces heat in the lines which means power is lost.
Power (heat) dissipation in the lines is given by P = VI where P is power, V is the voltage down the line, and I is the current in the line. So the same P can be due to high voltage times a small current, or low voltage times a high current. For power transmission over a long distance you want as high as voltage as possible.
For Edison, the technology was not there to increase the DC voltage. Westinghouse figured out how to build practical transformers. These transformers could step up the AC voltage and transmit power long distances with lower line losses than DC.
However, today's technologies allow the generation of HVDC (high voltage DC). Also, it ends up DC lines dissipate 30 to 40% less heat than AC lines, at the same voltage. That's because transmitted AC power has a "reactive" power component. This causes extra currents to flow in the transmission lines as needed to feed inductive and capacitive components within the loads. And also with DC, all of the sync problems with multiple AC generators on the grid go away.
If the world was starting from scratch today, we might well be using DC. HVDC is a growing niche today and is already a $10 billion market. For example, power is sent from offshore wind turbines to land using HVDC.
The reactive power component is just self inductance due to the time-varying field. AC also has higher losses due to the 'skin effect' of current being carried by the surface of the conductor, again because of the nature of a time-varying field.
Anybody else have one of those Maxwell's Four Laws T-shirts back in the day?
Good point, the skin effect makes AC transmission lines of the same diameter have higher electrical resistance than DC lines. Another plus for DC.
An EE friend's comment:
On a tour once I asked a power station guy about synchronizing a generator
basis of the question was how "springy " is the grid?
he said it's basically a rock of infinite inertia, said someone had once screwed up syncronizing a generator and it basically instantly moved to in sync with the grid
( yes virginia, impulse functions do exist)
.....and......
sheared off the 18" ( if i remembered that right, in any case BIG) diameter shaft to the turbine.
And in related news, EV's are famous for their high torque. Same physical principles, of course.
Yep. If the generator is phase-lagging when the breakers to the grid are closed, the term for what happens is ‘motorizing’ the generator.