William Vaughn

Psalm 106

Tue, 17 Jun 2025 13:04:00 -0700

Give Thanks to the Lord, for He Is Good

106 Praise the Lord!
Oh give thanks to the Lord, for he is good,
    for his steadfast love endures forever!
2 Who can utter the mighty deeds of the Lord,
    or declare all his praise?
3 Blessed are they who observe justice,
    who do righteousness at all times!

4 Remember me, O Lord, when you show favor to your people;
    help me when you save them,[a]
5 that I may look upon the prosperity of your chosen ones,
    that I may rejoice in the gladness of your nation,
    that I may glory with your inheritance.

6 Both we and our fathers have sinned;
    we have committed iniquity; we have done wickedness.
7 Our fathers, when they were in Egypt,
    did not consider your wondrous works;
they did not remember the abundance of your steadfast love,
    but rebelled by the sea, at the Red Sea.
8 Yet he saved them for his name’s sake,
    that he might make known his mighty power.
9 He rebuked the Red Sea, and it became dry,
    and he led them through the deep as through a desert.
10 So he saved them from the hand of the foe
    and redeemed them from the power of the enemy.
11 And the waters covered their adversaries;
    not one of them was left.
12 Then they believed his words;
    they sang his praise.

13 But they soon forgot his works;
    they did not wait for his counsel.
14 But they had a wanton craving in the wilderness,
    and put God to the test in the desert;
15 he gave them what they asked,
    but sent a wasting disease among them.

16 When men in the camp were jealous of Moses
    and Aaron, the holy one of the Lord,
17 the earth opened and swallowed up Dathan,
    and covered the company of Abiram.
18 Fire also broke out in their company;
    the flame burned up the wicked.

19 They made a calf in Horeb
    and worshiped a metal image.
20 They exchanged the glory of God[b]
    for the image of an ox that eats grass.
21 They forgot God, their Savior,
    who had done great things in Egypt,
22 wondrous works in the land of Ham,
    and awesome deeds by the Red Sea.
23 Therefore he said he would destroy them—
    had not Moses, his chosen one,
stood in the breach before him,
    to turn away his wrath from destroying them.

24 Then they despised the pleasant land,
    having no faith in his promise.
25 They murmured in their tents,
    and did not obey the voice of the Lord.
26 Therefore he raised his hand and swore to them
    that he would make them fall in the wilderness,
27 and would make their offspring fall among the nations,
    scattering them among the lands.

28 Then they yoked themselves to the Baal of Peor,
    and ate sacrifices offered to the dead;
29 they provoked the Lord to anger with their deeds,
    and a plague broke out among them.
30 Then Phinehas stood up and intervened,
    and the plague was stayed.
31 And that was counted to him as righteousness
    from generation to generation forever.

32 They angered him at the waters of Meribah,
    and it went ill with Moses on their account,
33 for they made his spirit bitter,[c]
    and he spoke rashly with his lips.

34 They did not destroy the peoples,
    as the Lord commanded them,
35 but they mixed with the nations
    and learned to do as they did.
36 They served their idols,
    which became a snare to them.
37 They sacrificed their sons
    and their daughters to the demons;
38 they poured out innocent blood,
    the blood of their sons and daughters,
whom they sacrificed to the idols of Canaan,
    and the land was polluted with blood.
39 Thus they became unclean by their acts,
    and played the whore in their deeds.

40 Then the anger of the Lord was kindled against his people,
    and he abhorred his heritage;
41 he gave them into the hand of the nations,
    so that those who hated them ruled over them.
42 Their enemies oppressed them,
    and they were brought into subjection under their power.
43 Many times he delivered them,
    but they were rebellious in their purposes
    and were brought low through their iniquity.

44 Nevertheless, he looked upon their distress,
    when he heard their cry.
45 For their sake he remembered his covenant,
    and relented according to the abundance of his steadfast love.
46 He caused them to be pitied
    by all those who held them captive.

47 Save us, O Lord our God,
    and gather us from among the nations,
that we may give thanks to your holy name
    and glory in your praise.

48 Blessed be the Lord, the God of Israel,
    from everlasting to everlasting!
And let all the people say, “Amen!”
    Praise the Lord!

I put the whole poem in because I think it’s a good recap of a lot of the old testament stories for those that don’t know it. The story of the Israelites is one of repeated brokeness and sin. They were sinned against as slaves in Egypt to the point that all they knew was a life of sin. God rescued them, because of his purpose to reveal himself to all the world through them. They struggle to break their cycles of sin, and God is patient, caring, and forgiving even though they don’t deserve it. Many people, even those who have tried to read the bible, will dismiss it because they find a lot of genuinely terrible things happening, and it can be difficult to discern when the bible is being descriptive vs prescriptive. This psalm is a reminder that God’s will for us is good, that he is the great cure for the disease of sin that breaks the world around us.

Verses 34-48 compelled me over the edge to post this. It occurred to me that one thing non-believers and atheists (I used to be one) don’t credit God for, is how his true followers created the values of the modern world they know. Non-believers will focus on the evils of religion that people have perpetrated in God’s name (which is undeniable, by the way). Christ would be the first to call out the hypocrisy, and we as followers of him have a duty to follow that example to build the world described in verses 1-5. However, if we forget or reject God’s love, and desire a world without it, it will will be given to us. That’s not the world we want, and it’s not a better world. Maybe that’s an explanation for what we see happening in our world now? These verses are a reminder that the world before God, and before Christ, was a place of unmitigated evil. Sacrificing and abandoning children was normalized or even required. Worship of pagan gods was often a corporate sexual worship (orgy) that created pain and brokeness in communities through a cycle of sex slavery, sex obsession, and sex abuse. These things should anger you, and they anger God. Thank God for that! Thank God and praise him that his love is abundant and steadfast to save us from ourselves, if we will let him. Will you let him?

Psalm 52

Mon, 02 Jun 2025 11:43:00 -0700

I’m still on the plan, but it has been so difficult to find time for all the reading and especially the blog writing in the midst of everything I have going on. Not mere business, but significant trials of health in our family. I find David’s laments in the Psalms heartening and his calls to trust fully in the Lord absolutely necessary. In the plan we’re in 1 Samuel, while also bouncing into the Psalms of David that match up to the story. This Psalm 52, is not a lament, it is a rebuke on Saul and Doeg the Edomite, while also being David’s declaration to wait on the Lord rather than depending on himself. It saddens me because it reminds me that there are so many people in this world who trust in themselves to their own destruction. I think of powerful people who through self-deception or malice love lies more than truth. I think of everyday people who trust lies the world tells them, and so never seek God as their protector and truth. When people trust in themselves, or in their success, their riches, their abilities, or anything other than God, what they have actually pursued is their own destruction.

The Steadfast Love of God Endures
To the choirmaster. A Maskil of David, when Doeg, the Edomite, came and told Saul, “David has come to the house of Ahimelech.”

1 Why do you boast of evil, O mighty man?
   The steadfast love of God endures all the day.
2 Your tongue plots destruction,
   like a sharp razor, you worker of deceit.
3 You love evil more than good,
   and lying more than speaking what is right. Selah
4 You love all words that devour,
   O deceitful tongue.
5 But God will break you down forever;
   he will snatch and tear you from your tent;
   he will uproot you from the land of the living. Selah
6 The righteous shall see and fear,
   and shall laugh at him, saying,
7 “See the man who would not make
   God his refuge, but trusted in the abundance of his riches
   and sought refuge in his own destruction!”
8 But I am like a green olive tree in the house of God.
   I trust in the steadfast love of God forever and ever.
9 I will thank you forever, because you have done it.
   I will wait for your name, for it is good, in the preseence of the godly.

The Holy Bible: English Standard Version (Wheaton, IL: Crossway Bibles, 2016), Ps 52:title–9.

I am praying for God to grant me His presence and to keep me close. I want to trust fully in Him even when it is difficult. I lack the diligence and discipline to do it, God must transform my habits, my life, and my heart to draw me in. Amen!

Leviticus 16:20-22, 17:10-16

Sun, 09 Mar 2025 09:53:00 -0700

20 “And when he has made an end of atoning for the Holy Place and the tent of meeting and the altar, he shall present the live goat. 21 And Aaron shall lay both his hands on the head of the live goat, and confess over it all the iniquities of the people of Israel, and all their transgressions, all their sins. And he shall put them on the head of the goat and send it away into the wilderness by the hand of a man who is in readiness. 22 The goat shall bear all their iniquities on itself to a remote area, and he shall let the goat go free in the wilderness.

ESV Leviticus 16:20-22

10 “If any one of the house of Israel or of the strangers who sojourn among them eats any blood, I will set my face against that person who eats blood and will cut him off from among his people. 11 For the life of the flesh is in the blood, and I have given it for you on the altar to make atonement for your souls, for it is the blood that makes atonement by the life. 12 Therefore I have said to the
people of Israel, No person among you shall eat blood, neither shall any stranger who sojourns among you eat blood. 13 “Any one also of the people of Israel, or of the strangers who sojourn among them, who takes in hunting any beast or bird that may be eaten shall pour out its blood and cover it with earth. 14 For the life of every creature is its blood: its blood is its life. Therefore I have said to the people of Israel, You shall not eat the blood of any creature, for the life of every creature is its blood. Whoever eats it shall be cut off. 15 And every person who eats what dies of itself or what is torn by beasts, whether he is a native or a sojourner, shall wash his clothes and bathe himself in water and be unclean until the evening; then he shall be clean. 16 But if he does not wash them or bathe his flesh, he shall bear his iniquity."

ESV Leviticus 17:10-16

Since the rebellion in Genesis 3, God has established that the necessary payment for sin, is death. Here in Leviticus, God is setting up a system of sacrifice and atonement for His people by which they must face the graphic consequence of their sin, but can also be forgiven of it. The exchange of blood, as the source of life, is the cost that God accepts for forgiveness. This sacrificial system established in Levitical law is pointing us to our need for an ultimate and perfect sacrifice that could cover all the sin of the entire world.

Though it is not in these particular verses, there is another goat that is bled to pay the debt of the Israelites. The one in these verses is the scapegoat. It carries the burden of sins for all the people, and it takes them away. It is cast out from the safety and protection of the camp and into the wilderness to die. Jesus was similarly cast out from His people when he was taken outside of Jerusalem to die. He carried the burden of the the world’s sin, and paid for it with his blood on a cross. His promise was to bring those who look to Him into a right relationship with God through His perfect sacrifice. Thank you, Lord! May the news of your goodness spread forever and call our hearts to rest in you!

Genesis

Thu, 13 Feb 2025 12:38:00 -0800

Let’s be real, there are a lot of places in the book of Genesis where you’re going to go “Ewww!”. Here’s a sample:

Cain murders Abel over simple jealousy. Ugh!

Something fairly undefined happens between Noah and one of his sons. It’s not clear what, but it has to do with one of them disrespecting him when they found him sleeping naked. Ewww?

When Lot is trying to prevent the men of Sodom from raping his house guests, he tries to distract them by offering to let them rape his daughters instead. They’re undeterred. Ugly on multiple levels!

After Sodom and Gomorrah have been destroyed, Lot’s daughters get him drunk and get themselves pregnant by him. Ewww!

Judah finds a wife for his son Er, her name is Tamar. Er is apparently evil, and God lets him die. Then Judah tells his other son Onan, “Okay, you go into her and have kids with her.” But Onan, uses the pull-out method to make sure he doesn’t get her pregnant. So God lets him die too. Tamar, a single, twice-widowed, childless, woman with no husband (basically a death sentence) is told to go back to her father’s house and wait until Judah’s youngest son, Shelah, is old enough to marry. When Shelah is old enough to marry, Tamar veils herself to get a look at him, and she’s wondering why she hasn’t been given to him in marriage yet. At that moment, Judah mistakes her for a prostitute and she goes along with it and becomes pregnant with twins! When it comes out that she’s been scandalous, she provides some proof of the prostitution incident and Judah realizes he was the one who messed up, and takes her for his own wife. Yeesh!

God makes a promise to be faithful to the children of Abraham and to bless the whole world through his descendants. All of his descendants though, they’re just normal people who do what normal people do. They mess up, make mistakes, do terrible things. But God, keeps his word and works it out the way he said he would anyway. That’s how Genesis goes for 40 chapters, until it ends with what I think is one of the most incredible stories in the Bible, the story of Joseph. This is where God works everything out for the good. Joseph’s brother’s get jealous of him (sort of like Cain did to Abel), they stop just short of killing him by selling him into slavery. They do evil to their brother, and they lie about it. God uses their evil to put Joseph second in command of Egypt. He gifts Joseph with insights and wisdom to prepare for a major famine, and in doing so he’s able to save the lives of thousands of people in and around Egypt, including the lives of his brothers. He reconciles with his brothers, and forgives them. Joseph fully yields to the goodness and provision of God. In the end of Genesis we find a picture of God’s grace and mercy. He brings all of the Israelites to Egypt to live and multiply for several hundred years until they’re ready to live out the next phase of his plan for them.

If you were to look at what the people say and do in Genesis as if it were the example to follow, you’d be right to question why it’s productive to believe in any of it. When I read this book as a non-believer, all it did was give me fuel to question even more why anyone would believe in this God. But when you instead focus on what God does and says throughout this story, you begin to see His character separately from the actions of people. If we’re to have a relationship with God as his children, we need to seek who He is rather than looking to the world for who they say He is. The world will show us hostility, immorality, hypocrisy, and violence and try to tell us that’s who God is. God shows us something else entirely.

Job 38:31-33; Job 39:9

Thu, 30 Jan 2025 10:23:00 -0800

Finally, God speaks. Thirty-eight chapters of Job’s misery and bad advice from his friends. God sure is patient. First order of business from God, is to establish who He is. Chapters 38 and 39 are really just God expounding on His glory and reminding Job and his friends who they are in relation to Him. He is the God that binds everything together, and sets all things in their order. One thing I find remarkable about these chapters is that they’ve stood the test of like 3000 years of discovery. All of the things God lists in these chapters that are his doing, are still things that with all our science and discovery, we still don’t fully understand. We have better understandings of them perhaps, and in some cases we’ve actually validated some of what God says in this chapter. But, mostly these were deep mysteries then, and they remain deep mysteries now.

31 “Can you bind the chains of the Pleiades
     or loose the cords of Orion?
32 Can you lead forth the Mazzaroth in their season,
     or can you guide the Bear with its children?
33 Do you know the ordinances of the heavens?
     Can you establish their rule on the earth?

ESV Job 38:31-33

Whoa. “Hey you, little human, did you set all this up? Did you create all these constellations and this whole universe, and work out exactly how it would all fit together and establish order from where you look upon them on earth?” The “Mazzaroth” is a Hebrew word, but from context it must be a constellation. Most Northern Hemisphere dwellers know the Pleiades, Orion, and The Bear (Ursa Major). I find it fascinating how human civilization has used the night sky for orientation, time keeping, navigation, myths, stories, and so much more. Here is God saying He, His Son, and His spirit worked out all that before time even began. God is big! I don’t want a God who is any less marvelous and mysterious. If we could understand the ways of God, He wouldn’t be worth worshipping. But, He is.

“Is the wild ox willing to serve you?
Will he spend the night at your manger?"

ESV Job 39:9

God continues in chapter 39, but now he’s covering the greatness of his command of all living creatures. He’s talking about knowing when and how the mountain goats give birth, and some stuff about sustaining wild donkeys in the salt lands. Then he puts in this little bit about how a wild ox would serve Him and spend the night by his manger. Do you know anyone born in a manger amongst barn animals? Yeah, maybe just maybe, in this book provably written thousands of years before it happened, God is giving a little hint at who He is, and what was to come. That Jesus, the living God would become human, be born in a manger to live a life we could never live, sacrificing himself for us so that we might have life and have it abundantly.

I really liked reading Job. This was my first time through it. Can’t wait to come back around to it next year! Perhaps in the time of the Pleiades.

Job 21:7; Job 22:17; Job 23:12-14

Tue, 21 Jan 2025 22:36:00 -0800

Why do the wicked live, reach old age, and grow mighty in power?

ESV Job 21:7

This is Job speaking in his reply to the bad advice his friends have been giving him. Basically, they say Job suffers because he sinned and isn’t right with God.

17 They said to God, ‘Depart from us,’
and ‘What can the Almighty do to us?’
18 Yet he filled their houses with good things—
but the counsel of the wicked is far from me.

ESV Job 22:17

Job knows God is just. He doesn’t fully understand why things are going so bad for him, but he knows and trusts that God is just. Ultimately, no one gets away with anything, but sometimes we want God to be immediate in carrying out justice, or to relent in our sufferings. We wonder, “Why haven’t you punished those that hate you, deny you, and work against you Lord? Why do people get away with being evil? Not only are they getting away with it, they’re thriving!” Job’s friends seem to think the world works like this: the wicked suffer, and the godly prosper. We see whole sects of religion subscribing to that theology, and preying on folks who want to believe it. The reality is that God allows us to be lost in our own evil, to depart from Him, and to have what we want; our world without Him. That can even go so well that we convince ourselves we don’t need God, and we’re in some sense our own God, in control of our own fate, our own standard for morality. It’s not what He wants for us, but He gives us the freedom.

Do we really want God to be immediate in His justice? I don’t know about you, but that wouldn’t end well for me. No, we rejoice that God doesn’t do what is “fair”, instead he gives us unimaginable mercy by saving us from what we deserve. He gives us his amazing grace, far more than we deserve, if we will turn to Him.

12 I have not departed from the commandment of his lips;
     I have treasured the words of his mouth more than my portion of food.
13 But he is unchangeable, and who can turn him back?
     What he desires, that he does.
14 For he will complete what he appoints for me,
     and many such things are in his mind.

ESV Job 23:12–14.

God does as he desires, he completes what he appoints. All manner of bad stuff is happening to Job, but the state of his heart is aligned directly toward God. God is always after our heart, and circumstances don’t always reflect the state of our heart.

Genesis 11:4-6

Tue, 14 Jan 2025 10:44:00 -0800

Then they said, “Come, let us build ourselves a city and a tower with its top in the heavens, and let us make a name for ourselves, lest we be dispersed over the face of the whole earth.” And the LORD came down to see the city and the tower, which the children of man had built. And the LORD said, “Behold, they are one people, and they have all one language, and this is only the beginning of what they will do. And nothing that they propose to do will now be impossible for them.

Genesis 11:4-6

Most people know this story as the “Tower of Babel”. God commanded the people to fill the earth several times (Gen 1:22,28, 9:1,7), and here they are puffed up on themselves, their own greatness, achievements, and self-sufficiency. They are planning to build a tower to reach heaven so they don’t have to be “dispersed over the face of the whole earth”. What I saw in the author’s character and God’s character here is humor. It’s hilarious, to me, that God has to come “down” in order to see their tower. Wasn’t this thing supposed to reach heaven? Even though it falls way short of the glory of God and the heavens, God still has the wisdom to look into their hearts for the true motivations for their pursuit. He calls it “only the beginning of what they will do”. I think what they “propose” to do is to replace their own need for God, and God is admitting that this proposal won’t be “impossible” for them to achieve. They’ll achieve it. We see that in our world now don’t we? Many people and cultures confident in their technology and own abilities. Powerful people puffed up on their own greatness, making themselves out to be more than they are.

Well, in this case God was working his plan to reveal himself to the world through Abram and the Jews, so he needed the people to disperse like he told them to do. Despite them trying again to mess up the plan, he confuses them with a language change, they tribe up and disperse over the earth exactly as planned.

Can I read the whole Bible this year?

Mon, 13 Jan 2025 20:18:00 -0800

It still blows my mind that I am a born-again Christian. If you knew me before, I’m sure you relate with that. If you think you’re weirded out by this change, imagine how I must feel! Truthfully, it is something shocking to think that even though I denounced and dismissed God (and anyone who believes in God) for so many years, He chose me anyway. I don’t fully comprehend all the things He weaved together to bring me to a moment of true repentance, where I finally comprehended the significance of words I had heard a thousand times, came to an end of insisting I was in control, and chose to believe something I’d spent most of my life denying. But, somehow He did. In an instant I knew in my heart that Jesus Christ had lived, said impossible things, did impossible things, willfully sacrificed Himself to erase a debt I owed, and then rose from the dead to prove that the payment was accepted. Crazy story right? When people ask, “Has anyone shared the Gospel with you?” this is it. The good news of a gift freely given and an opportunity to choose whether to dismiss it, or to accept it. This time I turned away from myself, toward a God who loves me, and I accepted. Doing that broke me. I started to sob uncontrollably. That word “uncontrollably” is for real, and it means I could not stop. It was kind of terrifying. I didn’t understand what was happening, but I look back now and understand it as a reaction to having the lies I’d centered my life around suddenly and drastically eroded. That moment everything changed. Every moment since has been different. In the moment that broke me, I admitted something that let Jesus in, and by letting him in He began the long work of changing me and building me back up. I admitted my actual helplessness. I admitted my own darkness. I admitted that there was a deep need for forgiveness in my life. That without forgiveness my life was lost, and that His forgiveness wasn’t something I could earn. It was something I had to accept.

Maybe you’re like me, you don’t want it. Maybe you’re like me, someday you will. I don’t know. But, that born-again moment for me was the beginning of having the word of God opened up. I had read parts of The Bible before, but just to poke holes in it. Now I wanted to read it, and when I did it met me where I was at and started to make sense.

The person without the Spirit does not accept the things that come from the Spirit of God but considers them foolishness, and cannot understand them because they are discerned only through the Spirit.

1 Corinthians 2:14

Since then, I’ve read a lot of The Bible. But this year, I am following a chronological plan of God’s story. I’m not going to post a lengthy post like this everyday, don’t worry. But, when a passage catches me, I will post it with a brief explanation of what it reveals to me about the character of God. If you’re here for it, my hope is that together we’ll understand God more fully – His purposes, His ways, His Mercy, and His love – by seeking what He has to say about these things Himself.

Emacs shell-command

Thu, 07 Jul 2022 00:07:00 -0700

When I used to use vim (or evil), I was able to use the :read command to execute any random shell command and plop its output into my current buffer. My main use case was in using base64 to encode secret values into Kubernetes Secret YAML manifests. It went something like this:

Go to the place in the buffer that I want to input a secret value.

:read ! echo -n 'Sup3rS3cr3t' | base64

And that would place U3VwM3JTM2NyM3Q= at the cursor, writing the base64 encoded value that was output from my shell command, executed from within vim.

Recently, I learned how to do this in emacs. It is M-!, or=shell-command=, to type in a command. And it is M-|, or shell-command-on-region, to run the shell command on the content of region (as STDIN).

You can use the C-u prefix to output the text resulting from the command to the buffer you run the command from.

Example, decoding a value.

C-u M-! enter echo 'Sup3rS3cr3T' | base64 the output is, U3VwM3JTM2NyM3Q=.

Now if you highlight that region, you can run C-u M-| input cat | base64 -d and it’s replaced with Sup3rS3cr3t again.

Thanks for reading, hope you enjoy this one neat trick you might not be using in emacs. All the best!

VP of Emacs: The Job

Sun, 15 May 2022 10:01:00 -0700

VP of Emacs is a blog series about using Emacs as an essential tool in my job as the VP of Software Engineering at Carbon Lighthouse. I’m a software engineer trying to cut it in a job that has required me to be more organized, communicative, strategic, and personal than ever. Emacs helps me get more done and keep it all connected. It has become the application I use every day to unify note-taking, emails, task management, writing, programming, and more.

This edition is a step back from Emacs and the details of my configuration. I wanted to talk about what my job is and how I approach it. I believe it is important to know what I’m trying to achieve and why, before a tool, even a tool as versatile as Emacs, can be used to its full potential.

People, Process, System, Strategy

A technical advisor of mine shared these four aspects of the VP of Engineering role with me, People, Process, System, and Strategy. When I heard them laid out like this, it resonated strongly with me. In a startup environment, the work of the VP role is often oscillating between developer, architect, product manager, coach, counselor, strategist, and diplomat. This should be expected, because the small team needs its leader to cover their gaps and be the glue that keeps things running smoothly. It is exhausting, though. At any given time, it is going to feel like one or more of these aspects is on fire. Even though they are all essential and important to the organization’s future success, they can’t all be equally important at any given time. I strive to be honest and forthcoming, both up and down the chain, about why I’m intentionally letting one aspect slip in order to focus on the others.

People

Supporting, hiring, and growing the skills of people in a software organization is essential for high performance. As a leader, I support people and I direct culture. My job is to provide an environment in which leadership, cooperation, accountability, and learning flourish. The behaviors that lead to better software for customers, and a lower cost of change, need to be the default behaviors of the organization. If doing the right thing can be easily resisted, it will be resisted. It’s my job to define what the right behaviors are.

Process

In every software company, there is a flow of work from idea to delivery. The production pipeline for code and the mechanisms of daily operations, are what I call “process”. Building a production pipeline can be as complex or more complex than the technological system and architecture it delivers. The organization’s lessons need to be constantly fed back into its process. Making sure the flow of critical information is present between the executive level, and every individual software engineer. When the executive level makes a shift that is out of step with reality at the engineering level, it is process which must amplify a feedback signal. Similarly, it is process which must ensure that what the engineers are doing is continually adjusting toward the strategic needs of the business and its customers.

System

The system is architecture, frameworks, tools, automation, infrastructure, and monitoring that must all work, and work well, in order for the team to build something customers continue to use. A technical leader needs to do more than understand the system. They have to guide and inspire change in it that leads to meaningful improvements in stability, malleability, velocity, and impact. Occasionally I do code-level spikes to understand, question, or measure what is good and bad about what we’ve already built or to discover new paths forward. People trust leaders who have done their job, or could do their job. If I lose touch with the work, I lose credibility too. There is a fine line here though, to not step over and do things that should be carried to completion by the team.

Strategy

It takes a lot to figure out the above three items, but it’s not enough. In this role, I’m the person that must be looking ahead to where the organization needs to go. Is there a business change coming soon that will require drastic changes? What are the next important improvements to make, and how do we sequence our growth to be where we want to be in 2 years time. What’s truly important? How do you communicate it and educate every person in the organization so they know it, feel it, and can act on it with intent?

Managerial Economics

I can’t go do all the things I want done, I have to bring others into my reasoning and empower them to carry it out in their own way.
Everything I do, that someone else could have done, was something I overpaid for while also robbing the team of a learning and growth opportunity. Double loss.
80% of the work is done by 20% of the team. This means high performers, not low performers, should receive the majority of my focus, coaching, and guidance.
Efficiency is accomplishing higher output with fewer resources. Productivity is doing more valuable work in the same unit of time. Delegation to the edges of the organization is what drives every member of the organization to focus on valuable work, while also increasing their capacity.

Hiring & Recruitment

Hiring well is the most important responsibility of any manager. No hire is ever perfect, but a bad hire can be truly catastrophic to culture, communication, and productivity. I could write an entire blog on this subject, but here are my brief guiding principles on the subject.

Don’t hire, delegate more.
Okay, you’ve decided to hire. Define the role, as well as the skills and behaviors that are necessary to perform in the role exceptionally.
Build your hiring and interview process around finding reasons to pass on candidates, not reasons to hire them.
Design your interview to investigate past behaviors and simulate the work of the role. There is no better predictor of future performance, than past performance.

How Emacs Helps

Emacs is my tool for thought and data gathering. It forms the connections between my ideas and lets me recall them when I need them. My org mode index/inbox file is where every task I need to keep on top of lands by default. Org-agenda reminds me of things I deferred and helps me prioritize. Working in a single app affords me the advantage of doing less context switching and having higher integration between my notes, tasks, and email. I enjoy the freedom of being able to evolve my system of work, limited only by what I can configure or write in elisp. Most importantly, Emacs makes the work more fun because I enjoy using it.

More in this series

VP of Emacs series

Thanks for reading. Please get in touch if you want to talk about engineering and leadership in your role, I’m happy to lend my perspective and learn from yours.

VP of Emacs: Meeting Notes

Sun, 13 Mar 2022 11:18:00 -0700

Watch the YouTube video that accompanies this blog.

Meeting notes with Org-roam dailies

I don’t often take exhaustive meeting notes. Many meetings, especially in the post-pandemic world, have shared online notes taken in something like Google Docs anyway. What I find useful in my personal notes is to capture who the meeting attendees were, any links to resources relevant to the meeting, a summary of some main talking points, and any action required items. The people I have meetings with will have their own page in my Org-roam system. Many of the projects, milestones, and subjects that my meetings are about also have their own roam notes page. The linking and back-linking features give me a way to connect my meeting notes with the people I talked to and the subjects we talked about. Taking these meeting notes in Org-roam dailies files further connects those things to when the discussion occurred.

Org-roam dailies are a daily journaling feature in Org-roam. They’re normal .org note files in a dailies/ directory, they have names like 2022-03-11.org. When I’m on an Org-roam page for a person, the majority of backlinks are to daily notes files. The backlinks become a record of when I talked to this person, and about what.

Navigating to daily notes

My Emacs configuration has the following keyboard shortcuts for navigating between Org-roam daily files quickly.

key coord	function
`SPC n r d t`	org-roam-dailies-goto-today
`SPC n r d y`	org-roam-dailies-goto-yesterday
`SPC n r d m`	org-roam-dailies-goto-tomorrow
`SPC n r d d`	org-roam-dailies-goto-date

Most often I’m using SPC n r d t as a quick means to snap back to the notes I have for the current day. In the morning, it is usually helpful to also flip to what happened yesterday with SPC n r d y. When the day is winding down, SPC n r d m helps me write things down that apply to my schedule tomorrow. In order to see what happened on a particular date, I will use SPC n r d d. It pops up a calendar buffer that can be used to navigate to any date you’re interested in.

Quick capture to daily notes

key coord	function
`SPC n r d T`	org-roam-dailies-capture-today
`SPC n r d Y`	org-roam-dailies-capture-yesterday
`SPC n r d M`	org-roam-dailies-capture-tomorrow
`SPC n r d D`	org-roam-dailies-capture-date

These commands will make a new entry in the target daily file but will not navigate to the buffer when complete. They are a good way to capture a note without leaving your current location. I use SPC n r d D to capture entries for a future date. If I have a presentation in a week, and I already have my speaking notes prepared, I’ll copy my speaking notes and use SPC n r d D / org-roam-dailies-capture-date to make an entry for the presentation on that day.

Meeting capture templates

Reiterating from above, what I find useful in my personal notes is to capture who the meeting attendees were, any links relevant to the meeting, a summary of some main talking points, and any action required to-do items. I find it helpful to use an org capture template for meeting notes to keep them consistent and easy to scan.

capture_templates/meeting.org

* %?
** Links
-
** Attendees
-
** Notes
** Summary
** ARs [0/1]
*** TODO

To use it, the template needs to be added to the org-capture-templates list.

(use-package! org
  :config
  (setq org-capture-templates
        '(("n" "Meeting note" entry
           (file buffer-name)
           (file "capture_templates/meeting.org")))))

My typical flow:

Jump to today’s daily file with SPC n r d t

Do SPC X n to capture an entry for the new meeting.

Select a capture template
=========================

[n]    Meeting note
----------------------------------
[q]    Abort

Write in the name and purpose of the meeting

Start doing Org-roam links for attendees and relevant topics

* My important meeting
** Links
   - https://some-link-to-whatever.com
   - https://docs.google.com/some-gdoc-for-meeting
** Attendees
   - Paul
** Notes
** Summary
** ARs [0/1]
*** TODO

Capture it into the dailies file with C-c C-c

Review and grooming

My days are full of meetings and I have to bounce from one to the next. There is no time to reflect on notes, clean them up for future use, or make sure I take action on the results. When I have 5 m, or at the end of a day, I review my daily notes to underline important parts, add connective links, and create action items in my to-do tracking inbox. When I adhere to this practice I find that a lot fewer things slip through the cracks and I get better use out of my notes later.

VP of Emacs: A Personal Wiki In Plain Text

Wed, 05 Jan 2022 10:40:00 -0800

VP of Emacs is a blog series about using Emacs as an essential tool in my job as the VP of Software Engineering at Carbon Lighthouse. I’m a software engineer trying to cut it in a job that has required me to be more organized, communicative, strategic, and personal than ever. Emacs helps me get more done and keep it all connected. It has become the application I use everyday to unify note taking, emails, task management, writing, programming, and more.

Watch the YouTube video that accompanys this blog.

A personal wiki with Org and Org-roam

A realization I had when I started doing software management was that I could no longer safely forget something and expect someone else to keep track of it. As a manager, I am the one expected to keep track of things. I don’t naturally multitask well, and when something is out of my sight it is well and truly out of my mind. If I were to have a chance at being a supportive and effective manager I knew I would need to find a way to augment my brain with something slightly addictive to use. When I first fired up an Org buffer I knew I had found a tool that appealed to my hacker nature in a way that would keep me coming back. Sometimes Emacs demands my attention and focus, I get sidetracked by a configuration tweak or technical fidgeting, but most of the time Emacs is a net-positive influence on my productivity.

Org is a plain-text file format for taking structured notes, as well as an Emacs mode that adds astonishing functionality for editing and interacting with .org files. Org is one of the “killer apps” for Emacs, it’s widely used and well supported. Most of my work revolves around Org in some way or another. If you haven’t ever used org-mode, I’d encourage you to get a feel for it.

Org Manual, a good place to go.
Org tutorials, curated org tutorial resources. Pick and choose what looks interesting to you, but I recommend beginning with the Rainer König video tutorials so you can see it in action quickly.
Evil Mode: Or, How I Learned to Stop Worrying and Love Emacs, is a conversion resource for vim lover.

Built on Org’s powerful base, lies Org-roam, a knowledge management system based on the features and approach of a proprietary research tool called Roam Research. Org-roam transforms a folder of disconnected .org files into a connected wiki and database of personal knowledge. Some people call this approach “building a second brain”, others refer to it as the Zettelkasten method. Whatever you call it, the result is being able to see connections between all of your notes that you couldn’t see before. Every page in roam has a list of “backlinks” to everywhere that page was mentioned. I’ll drive home the value of this as a software engineering manager with a simple but illustrative example.

At any time, I can go to the Org-roam page for my CEO to see the details of any conversation or meeting I have had with him, ever.

That doesn’t only apply to my CEO, that applies to everyone I work with. If we’ve had more than one conversation and done anything I might consider important business, you have an Org-roam page in my system.

Tools like Evernote, Todoist, Superhuman, and Roam Research are enough to do anything I’ll write about in this blog series, but there are benefits I like about this plain-text approach to note taking that you don’t get with any other app.

The data is mine. I own it. I store it. It’s easy to manage because it’s just a folder of text files.
Any tool that works with plain-text, works with my notes. I can cat out a note from the command line. I can run grep or rg to search over file names or content. I can encrypt it if I want to with gpg.
Storage space for plain-text files is minuscule.
Text is portable. If Emacs ceased to exist, I could still edit and read my notes just fine using any other text editor.

Everyone and every thing gets their own `org-roam` note

Everyone I work with has an Org-roam note that becomes the central hub of information I have about them. People are the most significant part of my system, but they aren’t the only thing I keep track of.

I have notes by topic:

Python
Management
Budgeting

I keep notes for particular projects:

2021 Budget
2021Q3 Software Engineer Hiring
Some Cleverly Named Internal Project
JIRA-123 TITLE

I have notes for books I read, blogs I frequent, etc:

Systemcrafters
Linux Command Line and Shell Scripting Bible
The Phoenix Project

And the departments in my Company:

Sales
Marketing
HR

The more completely I link related ideas, the better the graph of my knowledge becomes. I want to be able to see how all of these people, ideas, and resources relate to each other. The purpose of taking notes this way is to be able to recall and rediscover information when it is needed. The more linked, the easier my knowledge is to traverse. I try to set aside time a few minutes a day to quickly scan, groom, and link notes. I go deeper during my weekly review sessions, often moving notes around or inserting links between them.

Here is what one of my buffers might look like for one of my direct reports.

:PROPERTIES:
:ID:       aaaaaaaa-1111-2222-3333-bbbbbbbbbbbb
:ROAM_ALIASES: Jane
:END:
#+title: Jane Doe

Lead Software Engineer at Carbon Lighthouse

* Contact
- Phone: 555-1234
- Email: jane.doe@carbonlighthouse.com

* Personal
- Her son is John Jr 3yo
- She is an accomplished rock n' roll musician, drummer

* Performance Reviews
** 2021H2 Data Gathering...
** 2021H2 Performance Review...

* 2021
** 2021-09 September
*** 2021-09-15 Wednesday...
*** 2021-09-15 Wednesday...
*** 2021-09-29 Wednesday...
** 2021-10 October
*** 2021-10-06 Wednesday...

It contains contact information, some reminders of personal details, performance review notes, and a datetree of our one on one meetings. The one on one meetings are shown using a collapse feature of Org in order to hide their details. In addition to this information, the Org-roam buffer shows the list of backlinks to everywhere Jane has been linked. These two buffers alone give me a great jumping off point to contextualize how well Jane is doing in her role. I often have a record of feedback she’s given and received, links to project’s she’s working on, notes on her contributions to team meetings, and more.

One on one meetings

The one-on-one or “o3” is the most important meeting I have as a VP of Engineering. I schedule 30 minute meetings once a week with my direct reports and I have many bi-weekly, monthly or quarterly touch points with other people in my department and around the company. Even though these meetings are somewhat informal and loosely structured I still treat them seriously. I rarely cancel them, and if I do then I’m sure to reschedule promptly. Because these meetings are so meaningful I like to be reasonably prepared for them, but I don’t usually take more than 5-10 minutes to prep. When I know I have an o3 coming up with a person I will navigate to their Org-roam file and run a specialized o3 org-capture template to make a new entry in their datetree.

Here is the simple template org/capture_templates/o3.org with an agenda for me to fill in with my items before the meeting and to record anything that comes up during the meeting.

* Direct (10m)
* Manager (10m)
  - %?
* Future (10m)
* ARs [/]

If today is Wednesday, Jan. 05 and I navigate to Jane’s file in preparation for an o3, I can type SPC X o to automatically insert the template into that Jane’s datetree for today’s date.

* 2021
** 2021-09 September
*** 2021-09-15 Wednesday...
*** 2021-09-15 Wednesday...
*** 2021-09-29 Wednesday...
** 2021-10 October
*** 2021-10-06 Wednesday...
* 2022
** 2022-01 January
*** 2022-01-05 Wednesday
**** Direct (10m)
**** Manager (10m)
  - Type something here
**** Future (10m)
**** ARs [/]

The above can also be done on Monday, Jan. 03 in preparation for a meeting that happens Wednesday by typing SPC u 1 SPC X o. This version of the command prompts the user for a date before inserting the template into the file.

This is a capture customization of Emacs and Org that helps me do something specific to my particular workflow. All it takes is a little Emacs Lisp.

(use-package! org
  :config
  (setq org-capture-templates
        '(("o" "o3" entry
           (file+olp+datetree buffer-name)
           (file "capture_templates/o3.org")))))

This change sets some options that inform Org how to do all the heavy lifting. Maybe this is something you could imagine seeing in the preferences menu of your favorite note taking application. However, this is quite different than that because of what it reveals, that Emacs is configured by the elisp programming language. Emacs can be thought of as an elisp interpreter or framework for building applications, especially applications that involve text processing. In the long-term this is the most important feature of Emacs, it unlocks the power for every individual to make Emacs into whatever they want it to be. If that’s intriguing to you, I hope you’ll keep reading as I lay out more of the ways I am weaving my software organization management philosophy into Emacs.

Coming up next

Some of the things I have in mind to write more about are:

How I take meeting notes leveraging Org-roam’s “dailies” features, to have even more information about who I talked to, when, and about what.
The task management capabilities of Org and Org Agenda, and how I use them to store every idea that pops into my head, deferring them for later action without getting sidetracked.
Reading my email from Emacs with a tool called “notmuch”, and how I seamlessly integrate action items that arise from email with my other tasks. I can reference emails from any note to provide more context for anything I’m thinking about. All without ever leaving Emacs.
Sharing my notes in Google Docs, Markdown, PDF, HTML, or any other easily exportable target for the export features of Org.
How I connect to PostgreSQL to run queries from Emacs in order to generate metrics reports on my Team and our software product.
How I perform objective and data-centric performance reviews based on the information I can access about my team in Emacs and other tools.
And all the other things I learn how to do in 2022.

I hope I’ll see you around.

Moving to sourcehut

Sat, 04 Dec 2021 23:53:00 -0800

Sourcehut recently announced their new chat.sr.ht service to provide an IRC bouncer server for all paying members. You can read more about the launch on sourcehut’s blog. I’ve been running my own ZNC bouncer, but it broke and I haven’t got around to fixing it. As a result, I haven’t been using as much IRC. I’ve been looking to the release of chat.sr.ht as a way to get back into it. I use Emacs as my IRC client, and it took me one night of fiddling to get it working with the new Soju bouncer. The chat.sr.ht service requires login via SASL. The built-in ERC client in Emacs does not support SASL logins. I decided to use Circe instead. Below is a configuration that has been working well.

(use-package! circe
  :config
  (setq circe-network-options
        `(("Libera"
           :tls t
           :nick "nackjicholson"
           :host "chat.sr.ht"
           :port 6697
           ;; For security purposes pulling these from an encrypted secrets file
           :sasl-username "willvaughn/libera@my-client"
           :sasl-password "...sr.ht oauth token..."))
        circe-reduce-lurker-spam t))

(use-package! circe-color-nicks
  :config
  (enable-circe-color-nicks))

This service pushed me over the edge toward really trying sourcehut, and it’s great. I’m going to bring my projects onto the platform. This blog is now using https://srht.site/ for static web hosting. It was a painless migration. I’m planning to move many of my personal repos over to sourcehut. I will likely keep popular projects like aiosql where they are in order to not be too disruptive to the community already using them. One of the things that really attracts me to sourcehut is their business philosophy and uncompromising pursuit of being a truly open source first platform.

I also love their willingness to dispense with the status quo and push their users to change their habits and adapt to simpler and established tooling that works better for everyone.

I hope I’ll see your projects on sourcehut. Please come contribute on mine!

PyPA Python Packaging User Guide Review

Thu, 29 Jul 2021 19:26:00 -0700

Python Packaging User Guide

Python is incredible, but the packaging story is a pain point even for programmers with years of experience. These docs are worth reading slowly, allowing yourself to follow links and get a sense for what is at your disposal for package and distribution management. The core of this documentation is about how to use pip, setuptools, build, and python -m venv to great effect. Many of us aren’t as familiar with these basic tools as we should be. We’ve jumped away for abstractions that add layers of insulation between us and these basic tools before really trying in earnest to use them.

I used Poetry for a long time in my open source project aiosql, but after reading this PyPA documentation I switched to the basics. Someone should be able to download my code and run something as simple as pip install . or pip install -r dev-requirements.txt to get going. Using poetry was putting a burden on people who pick up the project and want to collaborate with me. I’m not saying not to use poetry. I think it has the best user experience and is the smartest of any tool available to the python developer. If your application team, working out of a private repository, has discussed and decided to align on using poetry and everything that comes with it, great. But, read the PyPA packaging guide anyway, because you might learn that the basic tools do what you want, are well supported, and don’t have as bad a UX as you thought.

Highlights

pip-tools is the minimal layer of extra tooling to pin dependencies for build reproducability that you actually need.
setup.cfg exists and should be used for as much as possible before you write a single line to a setup.py.
a good explanation of what sdist and wheel packages are.
Links to a lot of PEPs.
All the ways to deploy and distribute a Python application, with examples and trade-offs.

Thanks for reading, see you out there in the source code.

IRC Services at Home with k3s

Tue, 27 Jul 2021 19:47:00 -0700

This is a follow-on to my previous two blogs about running ZNC and Bitlbee in k3s/Docker.

In my prior k3s blog I ran a NodePort Kubernetes Service because I was unable to figure out how to get an Ingress setup and how to run ZNC behind a reverse proxy. I’ve figured that out now. It wasn’t entirely simple because there were a few non-obvious things that needed to be undestood.

IRC needs to be proxied via TCP not HTTP, and ZNC should take that traffic separately.
- ZNC Wiki - Revers Proxy
- Traefik IngressRouteTCP
Host Server Name Indication (HostSNI) only works if you’re using TLS, so routing by subdomains only works if you’re bothering with SSL/TLS.
- Traefik Endpoints
ZNC Configuration for URIPrefix allows ZNC’s webadmin to serve assets while behind a reverse proxy
- URIPrefix
The way to configure k3s addons like Traefik, is to drop their configuration in a server folder.
- k3s - Auto Deploying Manifests
Subdomains on a local network with mDNS is possible but not easy, and you should get on with your life. Edit your /etc/hosts my friend.
- 192.168.X.Y znc.vault.local bee.vault.local

Here’s what I have now. I’m using a Protectli Vault Mini-PC but I believe this whole thing could easily be run on a raspbery pi or whatever else you have lying around the house. :)

The steps described in detail in my previous two blogs will still be helpful in getting the services configured and initialized. Rather than repeat it all again, I’m going to share the YAML and let you explore. Feel free to ask me questions if you’re trying to setup something similar for yourself.

/var/lib/rancher/k3s/server/manifests/traefik.yaml

apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: traefik-crd
  namespace: kube-system
spec:
  chart: https://%{KUBERNETES_API}%/static/charts/traefik-crd-9.18.2.tgz
---
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: traefik
  namespace: kube-system
spec:
  chart: https://%{KUBERNETES_API}%/static/charts/traefik-9.18.2.tgz
  set:
    global.systemDefaultRegistry: ""
  valuesContent: |-
    rbac:
      enabled: true
    ports:
      bee-irc:
        port: 6665
        expose: true
        expoosedPort: 6665
        protocol: TCP
      znc-irc:
        port: 6667
        expose: true
        exposedPort: 6667
        protocol: TCP
      websecure:
        tls:
          enabled: true
    podAnnotations:
      prometheus.io/port: "8082"
      prometheus.io/scrape: "true"
    providers:
      kubernetesIngress:
        publishedService:
          enabled: true
    priorityClassName: "system-cluster-critical"
    image:
      name: "rancher/library-traefik"
    tolerations:
    - key: "CriticalAddonsOnly"
      operator: "Exists"
    - key: "node-role.kubernetes.io/control-plane"
      operator: "Exists"
      effect: "NoSchedule"
    - key: "node-role.kubernetes.io/master"
      operator: "Exists"
      effect: "NoSchedule"

traefik-ingress/base/routes.yaml

---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:
  name: znc-routes
  namespace: default
spec:
  entryPoints:
    - znc-irc
  routes:
  - kind: Rule
    match: HostSNI(`*`)
    services:
    - kind: Service
      name: znc-bouncer
      namespace: default
      port: 6667
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:
  name: bee-routes
  namespace: default
spec:
  entryPoints:
    - bee-irc
  routes:
  - kind: Rule
    match: HostSNI(`*`)
    services:
    - kind: Service
      name: bitlbee
      namespace: default
      port: 6667
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: web-routes
  namespace: default
spec:
  entryPoints:
    - web
  routes:
  - kind: Rule
    match: Host(`vault.local`) && PathPrefix(`/znc/`)
    services:
    - kind: Service
      name: znc-bouncer
      namespace: default
      port: 8888

bitlbee/base/service.yaml

---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: bitlbee
  name: bitlbee
spec:
  ports:
  - name: bitlbee-irc
    port: 6667
    protocol: TCP
    targetPort: 6667
  selector:
    app: bitlbee
  type: ClusterIP

bitlbee/base/pvc-bitlbee.yaml

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: bitlbee-local-path-pvc
  namespace: default
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: local-path
  resources:
    requests:
      storage: 1000Ki

bitlbee/base/pvc-signald.yaml

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: signald-local-path-pvc
  namespace: default
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: local-path
  resources:
    requests:
      storage: 1000Ki

bitlbee/base/deployment.yaml

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: bitlbee
  name: bitlbee
spec:
  replicas: 1
  selector:
    matchLabels:
      app: bitlbee
  template:
    metadata:
      labels:
        app: bitlbee
    spec:
      volumes:
      - name: bitlbee-data
        persistentVolumeClaim:
          claimName: bitlbee-local-path-pvc
      - name: signald-socket
        persistentVolumeClaim:
          claimName: signald-local-path-pvc
      initContainers:
      - name: volume-mount-hack
        image: ezkrg/bitlbee-libpurple:latest
        command: ["sh", "-c", "chmod -R 777 /var/lib/bitlbee && chmod -R 777 /var/run/signald"]
        volumeMounts:
        - name: bitlbee-data
          mountPath: /var/lib/bitlbee
        - name: signald-socket
          mountPath: /var/run/signald
      containers:
      - name: signald
        image: finn/signald:latest
        volumeMounts:
          - mountPath: /signald
            name: signald-socket
      - name: bitlbee
        image: ezkrg/bitlbee-libpurple:latest
        ports:
        - containerPort: 6667
        volumeMounts:
          - mountPath: /var/run/signald
            name: signald-socket
          - mountPath: /var/lib/bitlbee
            name: bitlbee-data

znc/base/service.yaml

---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: znc-bouncer
  name: znc-bouncer
spec:
  ports:
  - name: znc-irc
    port: 6667
    protocol: TCP
    targetPort: 6667
  - name: znc-webadmin
    port: 8888
    protocol: TCP
    targetPort: 8888
  selector:
    app: znc-bouncer
  type: ClusterIP

znc/base/pvc.yaml

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: znc-local-path-pvc
  namespace: default
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: local-path
  resources:
    requests:
      storage: 200Ki

znc/base/deployment.yaml

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: znc-bouncer
  name: znc-bouncer
spec:
  replicas: 1
  selector:
    matchLabels:
      app: znc-bouncer
  template:
    metadata:
      labels:
        app: znc-bouncer
    spec:
      volumes:
      - name: znc-config
        persistentVolumeClaim:
          claimName: znc-local-path-pvc
      initContainers:
      - name: init-config
        image: znc:1.8.2
        command: ['/bin/sh', '-c', "until [ -f /znc-data/configs/znc.conf ]; do echo waiting for znc.conf; sleep 2; done"]
        volumeMounts:
          - mountPath: /znc-data
            name: znc-config
      containers:
      - name: znc
        image: znc:1.8.2
        ports:
        - name: znc-irc
          containerPort: 6667
        - name: znc-webadmin
          containerPort: 8888
        volumeMounts:
          - mountPath: /znc-data
            name: znc-config

ZNC Webadmin Listeners

Running Bitlbee & Signald with Docker

Mon, 26 Jul 2021 02:48:00 -0700

Bitlbee is an IRC server which bridges to other common chat applications. I use it to text my friends on Signal from an Emacs ERC buffer. It’s possible to run bitlbee as a Docker container and systemd service on your personal Linux system, that’s what this blog will show you how to do.

Overview

In order for Bitlbee to work with Signal, you need some libraries, signald and libpurple-signald. The libpurple library is a common requirement for many bitlbee bridges, and fortunately there is already a suitable docker container with bitlbee and libpurple pre-installed, ezkrg/bitlbee-libpurple. However, it does not have signald installed in it. We will need to run another container, finn/signald, and give the bitlbee container access to it via a shared volume. Once the containers are set and available, we’ll use systemd to ensure they start on boot.

Setup Storage & Cross Communication Volumes

mkdir -p ~/.cache/docker-bitlbee/bitlbee
mkdir -p ~/.cache/docker-bitlbee/signald
chmod 777 ~/.cache/docker-bitlbee/bitlbee
chmod 777 ~/.cache/docker-bitlbee/signald

Run the Containers:

docker run \
    --name docker-signald.service \
    --detach \
    --restart=unless-stopped \
    -v ~/.cache/docker-bitlbee/signald/run:/signald \
    finn/signald
docker run \
    -p 6667:6667 \
    --name docker-bitlbee.service \
    --detach --restart=unless-stopped \
    -v ~/.cache/docker-bitlbee/bitlbee:/var/lib/bitlbee \
    -v ~/.cache/docker-bitlbee/signald/run:/var/run/signald \
    ezkrg/bitlbee-libpurple:latest

More official instructions are available at libpurple-signald HOWTO.md.

Connect your IRC client to the bitlbee container. To do that with ERC in Emacs I can evaluate the following elisp.

(erc :server "localhost" :port 6667 :nick "willvaughn")

Once connected, register the user.

register <password>

Then add the Signal bridge from the &root channel.

account add hehoe-signald +12223334444
account hehoe-signald set tag signal
account signal set auto-join-group-chats true
account signal set nick_format %full_name-sig
account signal on

This is going to fail and print out a qrencode command. The docker container doesn’t have qrencode installed it. Until that is fixed, you can work around it by installing qrencode on your system and running the failed command. The result will be a QR code image that you can scan in your signal phone app to verify the new signald device.

For myself on Arch Linux, this is what I did:

sudo pacman -S qrencode
qrencode -s 6 -o /tmp/signald_link_purple_qrcode.png 'tsdevice:/?uuid=yadayada'
xdg-open /tmp/signald_link_purple_qrcode.png

Once you scan the QR code witht he Signal app on your phone, you should receive a logged in message in your IRC client.

Finish configuration and enablement of the &signal account/channel. See bitlbee docs on Managing contact lists.

These are the channel settings I ran.

channel &signal set account signal
channel &signal set fill_by account
channel &signal set auto_join true
rename _12223334444 myname-sig

Save it (ends up in our shared docker volume so we don’t have to do this everytime we log in).

save

Create a Systemd User Unit

For more on how systemd works see Systemd/User - How It Works.

Make a signald service at .config/systemd/user/docker-signald.service

[Unit]
Description=A signald container service

[Service]
Restart=always
ExecStart=/usr/bin/docker start -a docker-signald.service
ExecStop=/usr/bin/docker stop docker-signald.service

[Install]
WantedBy=default.target

Make the bitlbee service at ~/.config/systemd/user/docker-bitlbee.service

[Unit]
Description=A Bitlbee IRC bridge container
Wants=docker-signald.service

[Service]
Restart=always
ExecStart=/usr/bin/docker start -a docker-bitlbee.service
ExecStop=/usr/bin/docker stop docker-bitlbee.service

[Install]
WantedBy=default.target

Inform Systemd about these unit changes.

systemctl --user daemon-reload

Start the IRC service.

systemctl --user  enable docker-signald.service
systemctl --user  enable docker-signald.service

Conclusion

That should do it. I hope this has helped you run bitlbee on your system.

Running ZNC Bouncer With K3s

Sun, 25 Jul 2021 14:15:00 -0700

In my previous blog My Personal IRC Renaissance I explained how and why I’ve recently started using IRC more. One of the things that I think keeps IRC relevant is its simplicity. The simplicity lends itself to hackability and experimentation. It’s great fun! The simplicity can also sometimes be a detriment. In our modern internet world we expect applications to store information and give it to us on-demand. IRC doesn’t work like that. It is as real-time as real-time gets. If you’re not logged in, you don’t see your messages. Running a ZNC Bouncer is one way to layer on some features to IRC that make it feel like a more modern application. ZNC stays logged into IRC for you, and you can connect to it from multiple IRC clients. The service buffes and replays messages you missed since your last connection.

For a while I was running ZNC on a Google Cloud Platform VM. Instructions and resources for how to do so are here, on the System Crafters Wiki. This solution worked for me for about a month, but since I’m not using that server for anything else it’s quite a price tag to pay just to run a ZNC service. There are lots of cheaper hosting solutions, but I decided instead to run ZNC inside my local home network where it’s not exposed to the broader internet. This is also inexpensive because I’m leveraging a Protectli Vault Mini-PC that I already own which is acting as the master node of a small Kubernetes cluster via k3s.

Implementation
Connecting from ERC

Implementation

Overview

Update: See my blog IRC Services at Home with k3s

Persistent Volume for ZNC Config

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: znc-local-path-pvc
  namespace: default
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: local-path
  resources:
    requests:
      storage: 200Ki

A Persistent Volume holds data after a deployment or pod is gone. It is used here to supply a permanent location for the /znc-data directory that the ZNC application works with. The ZNC webadmin itself interacts directly with a file /znc-data/configs/znc.conf. This is like the database for ZNC. It holds all information about your servers, nicknames, and configured options. Because this is a dynamic config, it changes as you use the ZNC web app, it would be a poor fit for a k8s configmap object. It needs to be updated while the pod runs. The PVC was the best solution I could come up with. Let me know if you come up with anything better!

ZNC Deployment

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: znc-bouncer
  name: znc-bouncer
spec:
  replicas: 1
  selector:
    matchLabels:
      app: znc-bouncer
  template:
    metadata:
      labels:
        app: znc-bouncer
    spec:
      volumes:
      - name: znc-config
        persistentVolumeClaim:
          claimName: znc-local-path-pvc
      initContainers:
      - name: init-config
        image: znc:1.8.2
        command: ['/bin/sh', '-c', "until [ -f /znc-data/configs/znc.conf ]; do echo waiting for znc.conf; sleep 2; done"]
        volumeMounts:
          - mountPath: /znc-data
            name: znc-config
      containers:
      - name: znc
        image: znc:1.8.2
        ports:
        - containerPort: 6667
        volumeMounts:
          - mountPath: /znc-data
            name: znc-config

Probably not the simplest deployment file you’ve seen. There’s a few things to go over here.

The volumes and volumeMounts are staking a claim on our pv disk space in order to store the /znc-data configs.
There is an initContainers section that defines a container named init-config, more about that below because it really ties everything together.
The znc container is straightforward, listening for connections on port 6667.

What’s up with init-config. Well, ZNC relies on you having a znc.conf in place before the service starts. It also has a command line option znc --makeconf that it expects you to run in order to interactively make that configuration file (and other important files). The init-config container is running a loop checking that the znc.conf file has been made and is in the persistent volume. The pod won’t complete it’s startup until znc --makeconf has been run. If, on the other hand the file has already been made, the init-config will be short lived.

The first time you start this deployment you’ll see something like this:

[nackjicholson@vault ~]$ k get po
NAME                           READY   STATUS     RESTARTS   AGE
znc-bouncer-XXXXXXXXXX-YYYYY   0/1     Init:0/1   0          84s

It is waiting for you to make the znc.conf. So how do you do that? Get in the container and do it yourself!

[nackjicholson@vault ~]$ k exec -it znc-bouncer-XXXXXXXXXX-YYYYY -c init-config -- ./entrypoint.sh --makeconf
[WARN  tini (64)] Tini is not running as PID 1 and isn't registered as a child subreaper.
Zombie processes will not be re-parented to Tini, so zombie reaping won't work.
To fix the problem, use the -s option or set the environment variable TINI_SUBREAPER to register Tini as a child subreaper, or run Tini as PID 1.
[ .. ] Checking for list of available modules...
[ ** ]
[ ** ] -- Global settings --
[ ** ]
[ ?? ] Listen on port (1025 to 65534): 6667
[ !! ] WARNING: Some web browsers reject ports 6667 and 6697. If you intend to
[ !! ] use ZNC's web interface, you might want to use another port.
[ ?? ] Proceed anyway? (yes/no) [yes]: yes
[ ?? ] Listen using SSL (yes/no) [no]:
[ ?? ] Listen using both IPv4 and IPv6 (yes/no) [yes]:
[ .. ] Verifying the listener...
[ ** ] Unable to locate pem file: [/znc-data/znc.pem], creating it
[ .. ] Writing Pem file [/znc-data/znc.pem]...
[ ** ] Enabled global modules [webadmin]
[ ** ]
[ ** ] -- Admin user settings --
[ ** ]
[ ?? ] Username (alphanumeric): myuser
[ ?? ] Enter password: MY_ZNC_PASSWORD
[ ?? ] Confirm password: MY_ZNC_PASSWORD
[ ?? ] Nick [myuser]: mynick
[ ?? ] Alternate nick [mynick_]:
[ ?? ] Ident [myuser]: mynick
[ ?? ] Real name (optional): First Last
[ ?? ] Bind host (optional):
[ ** ] Enabled user modules [chansaver, controlpanel]
[ ** ]
[ ?? ] Set up a network? (yes/no) [yes]:
[ ** ]
[ ** ] -- Network settings --
[ ** ]
[ ?? ] Name [freenode]: libera
[ ?? ] Server host (host only): irc.libera.chat
[ ?? ] Server uses SSL? (yes/no) [no]: yes
[ ?? ] Server port (1 to 65535) [6697]:
[ ?? ] Server password (probably empty): MY_LIBERA_PASSWORD
[ ?? ] Initial channels: #systemcrafters
[ ** ] Enabled network modules [simple_away]
[ ** ]
[ .. ] Writing config [/znc-data/configs/znc.conf]...
[ ** ]
[ ** ] To connect to this ZNC you need to connect to it as your IRC server
[ ** ] using the port that you supplied.  You have to supply your login info
[ ** ] as the IRC server password like this: user/network:pass.
[ ** ]
[ ** ] Try something like this in your IRC client...
[ ** ] /server <znc_server_ip> 6667 myuser:<pass>
[ ** ]
[ ** ] To manage settings, users and networks, point your web browser to
[ ** ] http://<znc_server_ip>:6667/

As soon as that’s done, the init-config containers original process sees the newly created /znc-data/configs/znc.conf file and kills itself.

ZNC NodePort Service

---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: znc-bouncer
  name: znc-bouncer
spec:
  ports:
  - name: znc-app
    port: 6667
    protocol: TCP
    targetPort: 6667
    nodePort: 32667
  selector:
    app: znc-bouncer
  type: NodePort

This is pretty straightforward, runs a NodePort service exposing port 32667 and forwards it to our deployment listening on the targetPort: 6667. From my diagram above this is what let’s us connect to the server using the http://vault.local:32667 URI.

I fiddled around with using an Ingress to do this, but IRC clients typically ask for a host and port i.e. znc.vault.local and I could not figure out how to get an Ingress and my local mDNS to allow me to find the vault using a URL like znc.vault.local. If you have found a better solution to this, please let me know, /query nackjicholson on Libera.

Update: I found a way to do this with Ingress, see IRC Services at Home with k3s.

Connecting from ERC

My go-to IRC client is ERC within Emacs, you can probably apply the same idea to whatever client you use.

One time:

(erc :server "vault.local" :port 32667 :nick "mynick" :password "myuser/libera:MY_ZNC_PASSWORD")

If you’re going to load that in your init.el you’d likely want something like this, and to leverage the auth-source module in emacs to load your password securely.

init.el

(defun my--start-erc ()
  (interactive)
  ;; ZNC
  (erc :server "vault.local" :port 32667 :nick "mynick"))

.authinfo.gpg

machine vault.local login mynick port 32667 password myuser/libera:MY_ZNC_PASSWORD

Conclusion

I truly hope you’ll find that running your own ZNC on Kubernetes is a great learning experience, and works quite well! Be well, see you on irc.libera.chat!

My Personal IRC Renaissance

Sat, 24 Jul 2021 19:29:00 -0700

Freenode->Libera + System Crafters

A few months ago the parts of the Internet concerned with free and open source software and IT were abuzz about an IRC Exodus from Freenode to Libera. I hadn’t used IRC in quite a while, and had never been a regular user. I’ve always been interested in it but had never consistently logged in. Early this year I began to occassionaly come across videos from David Wilson’s System Crafters YouTube channel in my social channels. The videos were excellent, informative, organized, and well-aligned with my interest in GNU Emacs and Linux at large. I really got hooked by the playlist on Learning Emacs Lisp. Just after the Freenode/Libera kerfuffle, daviwil (as I now know him) started the #systemcrafters channel on Libera and did an excellent video and live stream detailing how to configure one of the built-in IRC clients in emacs – ERC. I followed along and for the first time had an IRC client I loved using! I also found a community in the #systemcrafters channel that was welcoming, fun, helpful and collaborative.

Below you’ll find descriptions of how far I am down the rabbit hole. I honestly don’t know how much further it goes! 😄

ZNC

In the #systemcrafters channel I learned that other folks were enhancing their IRC experience by using a “bouncer server” called ZNC. IRC is an old protocol with its origins in the beginning of the world wide web, if you’re not logged in you will just miss the messages that were sent to you. The way around this is to keep your client connected all the time, or to run a bouncer server that stays connected for you. The ZNC bouncer has many features that end up making IRC a little more useable. Essentially, you connect to the ZNC service and it replays what you missed in all your channels since your last login.

I wanted to run my own ZNC. So, I dug in and figured out how to do it on my own Google Cloud instance running Centos 7. The culmination of that learning helped me write this addition to the System Crafter’s wiki: ZNC Bouncer Servers Setup. The only downside is that I was running a publicly exposed server on the internet and spending about $40/mo to keep it going. There are other more affordable hosting options of course, but I’ve decided to run it on a Mini-PC on my home network. There will be a blog coming soon on that! 🖥

Update: Check out the new blogs Running ZNC Bouncer with K3s and IRC Services at Home with K3s.

0x0.st

IRC is old, have I said that yet? Charmingly old mind you! To send files, images, or even some copy/pasta code users typically use file-sharing or pastebin services. One such service is https://0x0.st. And here’s how to use it from the command line.


HTTP POST files here:
    curl -F'file=@yourfile.png' https://0x0.st
You can also POST remote URLs:
    curl -F'url=http://example.com/image.jpg' https://0x0.st
Or you can shorten URLs:
    curl -F'shorten=http://example.com/some/long/url' https://0x0.st

After doing this two or three times, and having watched 7 videos about learning emacs lisp I decided I could write an emacs package to make this more convenient. I found that a package on sourcehut already existed ~pkal/nullpointer-emacs. Because my goal was really to learn Elisp, I read this package more than I used it. In doing so I found that I could fix a few bugs, refactor it for better testability, and add a few features to it. I reached out to pkal on Libera (yes!) and told him that I had made a fork. After some discussion we decided I should be the maintainer of the 0x0 namespace on MELPA. We transferred the project over to me. I’m officially an emacs lisp programmer! 🤓

willvaughn/emacs-0x0 on GitLab
0x0 on MELPA

Bitlbee & Signald

So, now I’m in ERC and IRC all the time. This is basically my facebook or instagram. It’s inevitable that I should start thinking about how I can ERC-ify all textual communication in my life. I use Emacs for email, RSS, youtube, reddit, and news. Why shouldn’t I use it for texting my friends and family? That’s what Bitlbee is for, it bridges IRC to other common chat applications. I’m now running my own bitlbee server on my local network too! Some nice folks wrote and distributed libpurple-signald, a library that bridges the Signal chat application to bitlbee, allowing me to text my wife and my friends from the lovely confines of emacs. 🗨

Update: Check out the new blogs Running Bitlbee & Signald with Docker and IRC Services at Home with K3s.

crafter-bot

I’m lurking on this project benoitj/crafter-bot, the friendly #systemcrafters IRC bot written in – you guessed it – Emacs Lisp! I haven’t contributed but I think benoitj is up to some interesting things and I’m keeping tabs on it. 🤖

Conclusion

Thanks for reading about my IRC revival. Come find me on the #systemcrafters channel on irc.libera.chat. You can /query nackjicholson anytime! Hope to hear from you soon! 🖖

The git-add manual review

Sun, 27 Dec 2020 23:26:00 -0800

The git-add command is easy to overlook, I’ve used it thousands of times without ever thinking about reading the manual. I was excited to find out what what was in here, because I use this command so much that any learnings will probably payoff quickly. From using magit in emacs I knew it was possible to surgically add lines and blocks of code into the commit index, but using the CLI I’ve never bothered to try. I typically only add full files.

Basics

This command exists to prepare the content of the “index” to be used for the next commit.

git add ./path/to/file.txt
git add .
git add --all

These are the ways I typically use this command, and they’re perfectly valid! This is about all you need to know in order to take your working tree changes and prepare them for a commit. If you need more control you can get into using wildcards:

git add docs/\*.txt

Pick and Choose with `--edit`

Use this with some caution, but this option on git add will let you change the diff directly to be what you want to apply to the index. It will show you the + and - signs at the front of lines and you can delete the signs or completely remove the lines. When you save the edit, the things you kept will be added to the index and the things you left behind will still be in the working tree.

Here’s a diff with two simple chunks where I added and deleted some lines from a README. With the --edit option I’d like to only commit part of these. With the verbosity of git diffs even this small example can be a lot to read, you should go edit some files and try it yourself to get a feel.

diff --git a/README.md b/README.md
index eb107d0..fa359f8 100644
--- a/README.md
+++ b/README.md
@@ -52,10 +52,6 @@ https://stackoverflow.com/questions/44527452/cant-open-lib-odbc-driver-13-for-sq

 #### Installing MySQL

-##### macOS
-
-TODO: Probably something like `brew install mysql` or `brew install mariadb`.
-
 ##### amzn2

 ```bash
@@ -63,6 +59,13 @@ sudo yum install -y https://dev.mysql.com/get/mysql57-community-release-el7-11.n
 sudo yum install -y mysql-community-client
 ```

+How about add some more description here.
+Oh but this is toooooo much info, so let's take it out
+blah
+blah
+
+I want to leave this important bit in though!
+
 ### Installing Project Dependencies

 ```

So I’ll edit that:

git add --edit

This opens up my $EDITOR (vim) and then I can tweak this diff. Here’s my edit that leaves the macOS header in the first diff chunk and important bits of the second chunk.

diff --git a/README.md b/README.md
index eb107d0..fa359f8 100644
--- a/README.md
+++ b/README.md
@@ -48,25 +48,28 @@ brew tap microsoft/mssql-release https://github.com/Microsoft/homebrew-mssql-rel
 brew update
 brew install msodbcsql mssql-tools
 ```
 https://stackoverflow.com/questions/44527452/cant-open-lib-odbc-driver-13-for-sql-server-sym-linking-issue

 #### Installing MySQL

##### macOS
-
-TODO: Probably something like `brew install mysql` or `brew install mariadb`.
-
 ##### amzn2

 ```bash
 sudo yum install -y https://dev.mysql.com/get/mysql57-community-release-el7-11.noarch.rpm
 sudo yum install -y mysql-community-client
 ```

+How about add some more description here.
+
+I want to leave this important bit in though!
+
 ### Installing Project Dependencies

 ```
 poetry install
 ```

 If you get a clang/ssl error during `poetry install`, it's possible your linker+brew+Apple aren't playing well together. Try pointing your linker to openssl directly, via:

Interactive additions with `-p`, `--patch`

Using the --interactive flag you get an interactive mode for exploring the state of your repo and what to add. The useful bit of this to me was the patch subcommand, and you can go to interactive patch mode directly using the --patch option instead of --interactive.

In patch mode you are presented with hunks of your diff and a prompt to decide what to do with each hunk. Here are the possible decisions:

prompt	description
y	stage this hunk
n	do not stage this hunk
q	quit; do not stage this or any remaining
a	stage this hunk and all remaining
d	do not stage this hunk or any remaining
g	select a hunk to go to
\/	search for a hunk matching regex
j	leave this hunk undecided, see next undecided
J	leave this hunk undecided, see next
k	leave this hunk undecided, see previous undecided
K	leave this hunk undecided, see previous
s	split the current hunk into smaller hunks
e	manually edit the current hunk

Pretty easy, answer the prompts and make your decisions. When you’re done the index will be updated with the hunks you selected. The ones you didn’t select are still unstaged.

Conclusion

I’ve been actively trying to make cleaner and smaller commits to simplify code review for my colleagues. I’m hopeful that what I learned here with git-add will help the next time I’m faced with a messy or complicated diff that could be split into simpler pieces. Thanks for reading!

Arch Linux on my Toshiba Chromebook 2 (2014)

Tue, 15 Sep 2020 08:20:00 -0700

I started by trying to install Fedora 32 on this machine, just could not figure it out. I don’t know if Fedora 32 Workstation is intended for use on this madly resource constrained little machine. So, the minimal nature of Arch Linux lends itself to having a chance of working!

Specs of the laptop

By looking at the Chrome OS Devices Arch Wiki I found some basic specs for the Toshiba Chromebook 2

Available	Brand	Model	Processor	RAM	Storage	Upgradable	SeaBIOS	Remarks
Jan 2014	Toshiba	CB30/CB35 Chromebook	1.4 GHz Intel Celeron 2955U	2GB DDR3	16GB eMMC	No	Yes	Custom firmware available

Prepping the Chromebook to be hacked on

This was fun. These resources helped me get it done. I flailed for a while and had to read these pages a lot in order to figure it out.

Installation process

https://wiki.archlinux.org/index.php/Installation_guide

Partitioning

I used parted to partition the disk. First I figured out which device was the correct one to partition by running…

lsblk -p

For me this was /dev/mmcblk0 with 14.7G of space. We’re working with a tiny SSD drive and that makes hard-drive space important. To simplify setup and because of the constrained disk space I decided to go with the required boot partition for UEFI and a single / root partition. For swap space, using a 1GB swapfile instead of a dedicated partition.

parted /dev/mmcblk0
(parted) mklabel gpt
(parted) mkpart "EFI boot partition" fat32 1MiB 261MiB
(parted) set 1 esp on
(parted) mkpart "root partition" ext4 261MiB 100%
(parted) quit

Formatting the partitions.

mkfs.vfat -F32 -n EFI /dev/mmcblk0p1
mkfs.ext4 /dev/mmcblk0p2

Mounting the new filesystem

mount /dev/mmcblk0p2 /mnt
mkdir /mnt/boot
mount /dev/mmcblk0p1 /mnt/boot

Mirror List

pacman -Sy reflector
reflector -c 'United States' -f 12 -l 12 --verbose --save /etc/pacman.d/mirrorlist

This didn’t work, so I just munged it locally. I’ll run it again after install.

Pacstrap

pacstrap base base-devel linux linux-firmware intel-ucode sudo vim emacs git networkmanager

The Rest

Follow the Arch Wiki Installation guide, and clicked on many links to read more. I used GRUB as a boot loader.

Post Install

Swap file

Arch Wiki - Swap File

I decided to use a 1GB swap file for swap space at /swapfile.

# dd if=/dev/zero of=/swapfile bs=1M count=1024 status=progress
# chmod 600 /swapfile
# mkswap /swapfile
# swapon /swapfile

And update etc/fstab

/swapfile none swap defaults 0 0

Use free -m to verify that the swap is available.

Manual pages

# pacman -S man-db man-pages

Conclusion

I just took apart a laptop, installed a new bios on it, and installed Arch Linux for the third or fourth time. Each time I get more comfortable with what a computer actually is every time I do this. I now have GNOME, firefox and Emacs installed on this little thing and I am actually using it around the house for basic end user style computing, and it works really well. Email me if you have one of these relics and want some help doing the same.

Installing CentOS on Protectli Vault

Wed, 05 Aug 2020 08:15:00 -0700

I am installing CentOS over the top of ubuntu on my Protectli 4-port Vault Mini PC.

Downloading

I downloaded a “minimal” ISO of centos from the CentOS Download page. The CentOS Wiki has these instructions that show I can copy this to my usb key with dd.

So, where is my USB key mounted?

lsblk -p

Mine is /dev/sdb.

sudo dd if=/home/nackjicholson/Downloads/CentOS-8.2.2004-x86_64-minimal.iso of=/dev/sdb

That takes a while, then you can eject it

sudo eject /dev/sdb

Shut down my vault, put the USB key in and follow the Protectli CentOS Install Docs.

Pi-hole, WireGuard, DNS-Over-HTTPS with a Raspberry Pi 3

Wed, 05 Aug 2020 08:12:00 -0700

Why?

RPI 3: Sitting around collecting dust.
Pi-hole: I hate ads, this is an amazing tool that blocks ads for all devices connected to a network.
DNS-Over-HTTPS: Fewer people having access to my internet usage information.
WireGuard: Secure remote access to my homelab, and with Pi-hole ad-blocking from my mobile phone.
Learning!

I am doing all of this to learn networking and linux administration. So, when I can choose the easy way or the hard way, I take the hard way. If you’re looking for the fastest way to setup a VPN and Pi-hole, you may want to consult another guide. This article is mostly for myself to have documentation of all the decisions I’m making at home. I will present the resources I consult alongside key artifacts and descriptions of where I varied my setup from the references.

Router specifics

A lot of what I did was enabled by my router, a capable tool called the EdgeRouterX. When relevant I’ll show the Firewall, NAT and DNS settings I used. You’re probably running something else, you may be able to do what I did with your router, or maybe not. I’ll leave it to you research how to make similar changes on your router or firewall.

Conceptual Diagram

Accessing my home network when I’m away, and getting the benefit of ad-blocking while I’m connected over the VPN.

SSH Access for the RPI 3

This is an optional step, you can plug a monitor into your RPI 3 if you want. I prefer to use my normal workstation and ssh into the RPI 3 to work on it.

When complete you should be able to SSH into the pi.

ssh pi@<your-pi-IP>

Running Pi-hole

The Pi-hole is a DNS server that you can use on your network to filter out ads before they every get to your browser page. This section details getting a Pi-hole up and running.

Pi-hole Docs

Prerequisites

Pi-hole Docs - Prerequisites

For firewall settings I blindly ran these commands after reading a little bit in the docs about what they do.

iptables -I INPUT 1 -s 192.168.0.0/16 -p tcp -m tcp --dport 80 -j ACCEPT
iptables -I INPUT 1 -s 127.0.0.0/8 -p tcp -m tcp --dport 53 -j ACCEPT
iptables -I INPUT 1 -s 127.0.0.0/8 -p udp -m udp --dport 53 -j ACCEPT
iptables -I INPUT 1 -s 192.168.0.0/16 -p tcp -m tcp --dport 53 -j ACCEPT
iptables -I INPUT 1 -s 192.168.0.0/16 -p udp -m udp --dport 53 -j ACCEPT
iptables -I INPUT 1 -p udp --dport 67:68 --sport 67:68 -j ACCEPT
iptables -I INPUT 1 -p tcp -m tcp --dport 4711:4720 -i lo -j ACCEPT
iptables -I INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

ip6tables -I INPUT -p udp -m udp --sport 546:547 --dport 546:547 -j ACCEPT
ip6tables -I INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

Installing

Pi-hole has TUI application to aid installation.

curl -sSL https://install.pi-hole.net | bash

My answers to the questions the TUI asks:

Note that throughout this article 192.168.1.1 is my router gateway, and 192.168.1.63 is the IP of my RPI 3.

Select Upstream DNS Provider. To use your own, select Custom.
- Cloudflare
Pi-hole relies on third party lists in order to block ads.
- StevenBlack
- MalwareDom
Select Protocols (press space to toggle selection)
- IPv4
- IPv6
Do you want to use your current network settings as a static address?
- IP address: 192.168.1.63/24
- Gateway: 192.168.1.1
- Yes
Do you wish to install the web admin interface?
- On
Do you wish to install the web server (lighttpd)?
- On
Do you want to log queries?
- On
Select a privacy mode for FTL.
- 0 Show Everything

After install be sure to copy your admin webpage login. I also checked the lines added to /etc/dhcpcd.conf (as referenced in the prerequisites). I didn’t actually find that Pi-hole had added any lines, so I configured dhcpcd.conf myself to statically set an IP for the RPI 3 on my network.

# Static IP configuration:
# Using my EdgeRouterX 192.168.1.1
# Pointing the nameserver locally and backed up with Cloudflare
interface eth0
static ip_address=192.168.1.63/24
static routers=192.168.1.1
static domain_name_servers=127.0.0.1

In EdgeRouterX I configured the LAN settings to have a reserved IP (Static MAC/IP Mapping) for the RPI 3.

Name	MAC Address	IP Address
rpi3	xx:xx:xx:xx	192.168.1.63

Use Pi-hole for DNS with EdgeRouterX

With this configuration every device in the network uses the Pi-hole server to resolve DNS names.

Whitelist management script

This section details setting up a script to update the Pi-hole with commonly whitelisted domains. With this setup I rarely find that I need to whitelist random things that shouldn’t be blocked by the Pi-hole.

anudeepND whitelist

git clone https://github.com/anudeepND/whitelist.git

I setup a cron job with crontab -e

@weekly sudo /home/pi/whitelist/scripts/whitelist.sh

DNS Over HTTPS

Doing DNS over HTTPS adds another level of privacy, but it’s definitely optional. I chose to do this for my own learning.

Pi-hole Docs - DNS over HTTPS

Most, if not all, of this is in those docs. I’m just putting it here for documenting files I made on the RPI 3.

wget https://bin.equinox.io/c/VdrWdbjqyF/cloudflared-stable-linux-arm.tgz
tar -xvzf cloudflared-stable-linux-arm.tgz
sudo cp ./cloudflared /usr/local/bin
sudo chmod +x /usr/local/bin/cloudflared
cloudflared -v
rm ./cloudflared
rm ./cloudflared-stable-linux-arm.tgz

Configuring cloudflared to run on startup

sudo useradd -s /usr/sbin/nologin -r -M cloudflared
sudo vim /etc/default/cloudflared

# Commandline args for cloudflared
CLOUDFLARED_OPTS=--port 5053 --upstream https://1.1.1.1/dns-query --upstream https://1.0.0.1/dns-query

sudo chown cloudflared:cloudflared /etc/default/cloudflared
sudo chown cloudflared:cloudflared /usr/local/bin/cloudflared

sudo vim /etc/systemd/system/cloudflared.service

[Unit]
Description=cloudflared DNS over HTTPS proxy
After=syslog.target network-online.target

[Service]
Type=simple
User=cloudflared
EnvironmentFile=/etc/default/cloudflared
ExecStart=/usr/local/bin/cloudflared proxy-dns $CLOUDFLARED_OPTS
Restart=on-failure
RestartSec=10
KillMode=process

[Install]
WantedBy=multi-user.target

sudo systemctl enable cloudflared
sudo systemctl start cloudflared
sudo systemctl status cloudflared

Test it:

dig @127.0.0.1 -p 5053 google.com

And configure the Pi-hole to use it as its Custom IPv4 Upstream DNS 127.0.0.1#5053. Below are my complete Pi-hole DNS settings.

Exposing our RPI 3 to the internet with DDNS.

The Engineer Workshop blog describes using ddclient to set up a dynamic DNS client to inform the DNS nameservers of the world automatically whenever our dynamic IP is changed by the ISP. My EdgeRouterX can do this instead, so I opted to go that route. I use Namecheap as my registrar so there are some screens here of it’s administration UI.

Namecheap and EdgeRouterX DDNS setup

All of the Namecheap settings you need to change will be under the Advanced DNS on the management page for your domain. I setup DynamicDNS in my Namecheap domain management interface. There is a simple toggle to enable it and you get back a password. What DynamicDNS does is allow a DNS client to update the IP where the domain name mydomain.com resolves to. My EdgeRouterX has ddclient installed on it and has a configuration screen for it. The router can detect when my ISP has changed my Public IP and it will automatically update my domain name with the new IP.

To this end I setup an A+ Dynamic DNS Record for my top level domain, using 127.0.0.1 as a default value to begin with. If we setup our EdgeRouterX/ddclient correctly we should see this get updated with the real PublicIP of our home network.

Type	Host	Value	TTL
A+	@	127.0.0.1	Automatic

Here is the EdgeRouterX configuration:

The URL field there is dynamicdns.park-your-domain.com/getip. And the Server is dynamicdns.park-your-domain.com

EdgeRouterX Port Forwarding

The last step in exposing our server to the internet is to punch a hole in our firewall that allows traffic to our domain name to be forwarded to port 51820 on our RPI 3 where the WireGuard VPN will be running.

You won’t know if this is fully working until after you get through setting up WireGuard. You can verify what you have by instead opening up port 80 breifly and seeing that you can reach the Pi-hole webapp. BE SURE TO CHANGE IT BACK

Running WireGuard

WireGuard is a new-ish VPN technology that attempts to be simple and secure.

Installing

I followed the instructions on this github page . They were also referenced in this ]] which I also followed. It is fairly involved because wireguard isn’t in the main raspian distro. I chose to follow a mix of these instructions, but ended up writing my keys to a restricted location using umask 077 for a little extra security. I also chose to make my wg0 interface IPs be in the range 192.168.2.0/24. My RPI 3 WireGuard Server is 192.168.2.1 and it’s peer (the client) is 192.168.2.2. I also used the wg-quick@wg0.service systemd unit to run the server.

Here is my server config /etc/wireguard/wg0.conf.

[Interface]
Address = 192.168.2.1/24
SaveConfig = true
PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -A FORWARD -o %i -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -D FORWARD -o %i -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE
ListenPort = 51820
PrivateKey = <server_private_key>

[Peer]
PublicKey = <client_publick_key>
AllowedIPs = 192.168.2.2/32

Setting up my Arch Linux client

I’m using the linux-lts kernel so I ran the following.

# pacman -S wireguard-tools wireguard-lts systemd-resolvconf

And:

# systemctl enable systemd-resolved.service
# systemctl start systemd-resolved.service

My /etc/wireguard/wg0.conf

[Interface]
Address = 192.168.2.2/24
PrivateKey = <client_private_key>
DNS = 192.168.2.1

[Peer]
Endpoint = mydomain.com:51820
PublicKey = <server_public_key>
AllowedIPs = 0.0.0.0/0, ::/0
PersistentKeepalive = 21

To use the connection:

# wg-quick up wg0
# ... do things ...
# wg-quick down wg0

You can also download the app for WireGuard on your phone and connect with it. You can generate a QR code to scan on your phone here.

Conclusion

I hope you got through it too! My ads don’t always block when I’m on my Arch Linux laptop unfortunately. Be sure to email me if you find the solution to that. Thanks for reading!

Formatting and reinstalling raspian on Raspberry Pi 3

Sun, 02 Aug 2020 11:32:00 -0700

I am repurposing one of my Raspberry Pi’s and need to format the microSD card. I did this from my arch linux laptop.

Formatting the disk with `parted`

I started from the official Raspberry Pi Install Docs and that eventually linked me out to this blog by Norman Dunbar. I looked through it but decided to follow this guide using parted to format the disk from the command line. I came back to Norman Dunbar’s blog later on for instructions on loading NOOBS onto the microSD in prep for installing raspian.

# parted /dev/sda --script -- mklabel msdos
# parted /dev/sda --script -- mkpart primary fat32 1MiB 100%
# mkfs.vfat -F32 /dev/sda1
# parted /dev/sda print
Model: Generic- SD/MMC (scsi)
Disk /dev/sda: 31.9GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start   End     Size    Type     File system  Flags
 1      1049kB  31.9GB  31.9GB  primary  fat32        lba

Loading an Image onto the microSD

I looked into the official instructions for installing boot images onto the microSD here. Those instructions are complicated and involve using dd a whole lot. I decided I really don’t want to brick my laptop and I should use NOOBs. The Raspberry Pi docs link to this blog by Norman Dunbar for loading NOOBs onto the disk, it’s pretty easy.

Verify the sha1 of the downloaded NOOBS zip is the same as the one on the downloads page.

$ sha1sum ~/Downloads/NOOBS_v3_4_0.zip

Find where the disk is mounted.

$ mount | grep -i sda

For me, this informed me that I needed to go here to extract the NOOBS zip.

$ cd /run/media/nackjicholson/571C-B47B
$ unzip ~/Downloads/NOOBS_v3_4_0.zip
$ cd ~

Unmounting the disk.

# unmount /dev/sda

Done! This microSD is ready for the pi.

Installing Raspian with NOOBS

Start up the Raspberry Pi 3 with the NOOBS disk, follow the easy to use application to install an OS.

The git-diff manual review

Sun, 22 Mar 2020 17:07:00 -0700

This is a fairly short, but informative, manual for a powerful command in the git arsenal. At its simplest the manual explains the following uses of git-diff:

Show changes between the working tree and the index (or another tree).

AKA What you could tell git to add to the staging area for a commit.
```
git diff
```
Show changes between the index and tree.

AKA Show what is staged for the next commit.
```
git diff --cached HEAD
```

Show changes between two trees.

AKA Show difference in between commits

# Show diff for the last few commits to where our branch is now.
git diff HEAD~3 HEAD

# For ranges
git diff 9a29ab57 HEAD
git diff 9a29ab57..HEAD

Show changes between two blob objects.

AKA diff specific files.

# Show changes to the config.toml file in the last 3 commits.
git diff HEAD~3..HEAD -- ./config.toml

Show changes between two files on disk.

AKA Show difference in your working tree, to what’s in a commit.
```
# What has changed about these files.
git diff HEAD -- ./config.toml
git diff HEAD~3 -- ./blog.org
```

The way I wrote these here is close to how they’re written in the manual. Understanding git-diff is aided by understanding the flow of git commits and objects; concepts like the index (staging area), the working tree, and references. I suggest reviewing the gittutorial manuals to brush up on these concepts if you need to.

Some other highlights of the manual:

In depth examples of diff output formats.
Options like --stat and --shortstat for at a glance change information.
--ignore-all-space for ignoring whitespace changes.
in the weeds info about diff algorithm options.
Some options that really only exist for writing scripts against.

Thanks for reading this manual review! Be well.

Arch Linux LTS Kernel

Sat, 04 Jan 2020 18:59:00 -0800

Preparation

In order to stop my system from continuing to update the original linux kernel, I edited the ignored packages list in my /etc/pacman.conf file.

IgnorePkg   = linux

Then I ran an update to the system.

sudo pacman -Syu

Installing the LTS Kernel

Using pacman to install the LTS kernel package.

sudo pacman -S linux-lts

I’m using a standard boot loader that stores configs in /boot/loader/loader.conf. I needed to inform my boot loader configuration to load the new LTS kernel I just installed when booting up. My loader.conf points at another config at /boot/loader/entries/arch.conf which I needed to edit the following lines.

/boot/loader/entries/arch.conf

title Arch Linux (ENCRYPTED)
linux /vmlinuz-linux-lts
initrd /initramfs-linux-lts.img

The final step was to reboot my system.

reboot

Once my machine was restarted it was using the linux-lts kernel.

uname -rs
# Linux 4.19.92-1-lts

The tmux manual review

Wed, 04 Dec 2019 17:26:00 -0800

The tmux manual is extensive, well written, and informative. It is an excellent reference document, but I don’t think it is particularly well-suited to teaching the concepts of tmux and how those concepts can be used as part of a user’s personal workflow.

The most useful thing I can say to the uninitiated person curious about tmux is:

Tmux does for terminal windows what your Desktop OS does for application windows. It helps you organize, partition, and navigate your terminal windows in a way that allows you to background tasks and switch your focus effectively.

If you consistently have more than two terminal windows or tabs open, you can benefit from the hour or so it takes to learn the basics of tmux. Here are a few links to get you going.

Tao of Tmux
tmux manual
tmuxp session manager

You are also welcome to use my .tmux.conf as inspiration for your own. It is available here.

Concepts

Here is the hierarchy of the tmux system.

server
- sessions
  - windows
    - panes

When “attaching” to tmux, you do so as a client of the server. The server contains multiple named sessions which it can share amongst clients. Each session has a collection of windows and panes. For me, sessions are how I organize my projects. I usually have a tmux session available for each project I’m working on. Switching sessions equates to switching projects. This way I’m focusing on one project/session at a time, but can easily switch to another when I need to without losing my progress. Within a session, I keep multiple windows that help me organize the different subtasks I have to do related to the project. This could be things like starting a webserver or a text editor. Finally, panes are for viewing two terminal instances next to each other on the same screen. You can have your text editor open on the left pane, and the command line open on the right pane.

The Tao of Tmux book has some excellent visualizations which help clarify this organization of terminal instance.

Basics of tmux

The prefix

The prefix is a key combination which signals that you want to do something within tmux. Before typing a tmux command, you always hit the prefix first. The default prefix is C-b.

Essential commands

Once you’ve hit the prefix, then you can hit another key or set of keys to perform an action in tmux.

command	description
`?`	Show all the available tmux commands
`s`	Opens menu to switch sessions
`d`	Detach from session
`c`	Create a new window
`n`	Next window
`p`	Previous window
`%`	Vertical split window into panes
`"`	Horizontal split window into panes
`Up` `Down` `Left` `Right`	Navigate panes
`x`	Close pane (with confirm)
`&`	Close window (with confirm)
`,`	Rename window
`:`	Execute commands
`swapw -s 1 -t 0`	Swap windows
`neww -c <path> -n <name>`	Create a new window and set its name and cwd
`[`	Enter copy mode

See the manual for more about copy mode. One way I use this copy mode is to dump all of my interesting terminal output to a file that I want to save or share.

Enter copy mode Prefix+[
gg to go to the top of the file
V to enter visual block mode
G go back down to the last line
M-w Copy the text to buffer (kill-ring) (I have this as bound to y for obvious reasons)
Prefix+: and :save-buffer ~/path/to/output.txt

About session managers

There are a few session manager tools out there for tmux, and all of them address the central issue of being able to save tmux session configurations. It can be a drag to restart your computer and then try to tmux at -t my-project only to find that all of the windows and panes that you so carefully created have been annihilated. Using a session manager can help alleviate that pain. My personal favorite is tmuxp because it is written in python and has the tmuxp freeze feature to export the current session to a file. Session managers are not required, you don’t have to install one right away. Go try tmux for a while, if you find that you get annoyed when you lose a session – install a session manager!

Conclusion

I can’t emphasize enough that you should take a few hours to read The Tao of Tmux book. It is a much more thorough walkthrough of tmux concepts than I gave here. It has illustrations and examples that will illuminate the ways you can use tmux to do whatever fits your workflow. Best of luck on your journey with tmux! Thanks for reading!

Emacs wisdom from the internet

Sat, 30 Nov 2019 15:25:00 -0800

This is a collection of useful things I learned while searching the web for how to do something in Emacs. This blog is my effort to better remember and share these lessons.

The basics of Emacs copy/paste

Start visual selection C-SPC then navigate to end of selection.
M-w puts that text in the kill ring without deleting it. (Copy)
C-w puts that test in the kill ring and deletes it. (Cut)
C-y will “yank” that text out of the kill ring back into the buffer. (Paste)

Org Mode

Sparse tree search and tags

https://orgmode.org/manual/Sparse-trees.html https://orgmode.org/manual/Tag-Searches.html#Tag-Searches

C-c / m then the tag value.

Exiting a sparse tree is kind of weird but can be accomplished with C-c C-c and some Shift+Tab to cycle visibilities.

Narrowing Scope

Learn more: here

C-x n s To narrow a org mode to items under a single header

C-x n w To widen it back out. :praise:

Date and Time insertion

C-c ! Insert the current date

C-u C-c ! Will get you the current date and time!

Archiving `DONE` items

Learn more: here

C-c C-x C-a Archive a single item.

C-u C-c C-x C-s. Prompts for archives of items with status DONE. This can also be shortened to C-u C-c $.

Adding a CREATED property timestamp to an org item after it was made

Add the CREATED property C-C C-x p , choose created, nil
Go edit it and do C-u C-c ! to insert the timestamp.

VIM/EVIL

Selecting and changing quoted strings.

Learn more: here

va" visually select string including the quotes.
vi" visually select the content of the string.
ci" change the content of the string, between the quotes.

Substitution

Replace from the current line to the end of the file: :.,$s/target/replacement/g

Replace in the region all occurrences: :'<,'>s/target/replacement/g.

You can do this by visually selecting and then pressing : to enter ex mode.

Motion

Remember to move around with { and }. Moving by code blocks/paragraphs.

Ivy Mode

C-M-j Don’t use the selection, just enter the text you’ve written!

man

M-x man Opening man pages.

ispell spell check

brew install ispell

M-x ispell to run the spell checker. Press A to accept a word for session. Use numbers in prompt at the top of the screen to select the suggestions.

Undo Tree

C-_ is undo. You can visualize the tree of things you undo. It’s C-x u. You can then move around that buffer, and the buffer you were editing gets previewed at each place in the undo tree.

Info mode

I cloned https://github.com/webframp/sicp-info and then I can read it in emacs by opening the .info buffer and running:

M-x Info-on-current-buffer

Python Debugging

Be in a file you want to run the debugger from. Or you can use M-x cd to change your working directory. To start the emacs pdb integration type M-x pdb.

This will prompt you in the minibuffer with:

Run pdb (like this?): pdb

You want to replace that pdb part with:

Run pdb (like this?): python -m pdb your_code.py

Run pdb (like this?): python -m pdb -m module.path.to.your_code

That is going to kick you into the pdb prompt. Now you need to type help and read the pdb docs.

Arch Linux First Steps

Sat, 30 Nov 2019 13:41:00 -0800

This follows my last post Arch Linux on my Lenovo t470s with more details about my own personal setup with Arch Linux. I use GNOME and really like it, if you’re new to Linux I think it’s a comfortable entry into the space, and if you’re not new it’s still extremely ergonomic, capable and comfortable.

Enable system NetworkManager

systemctl enable NetworkManager
systemctl start NetworkManager
nmtui # terminal ui to activate and connect to wifi

Install `zsh`

pacman -S zsh

User and groups setup

https://wiki.archlinux.org/index.php/Users_and_groups#User_management

useradd -m -g wheel -s /usr/bin/zsh <USERNAME>
passwd <USERNAME>

Go edit the sudoers file with EDITOR=vim visudo and comment out the %wheel line. It will add all users in the wheel group to the sudoers.

Install `gnome` and `gnome-extra`

https://wiki.archlinux.org/index.php/GNOME

pacman -S gnome gnome-extra

Enable gdm https://wiki.archlinux.org/index.php/GDM

systemctl enable gdm.service
reboot

Should see the GNOME login managed by gdm.

Now in order for ``totem`` to work aka GNOME Videos app to play H256 mp4 vids install ``gst-libav``

sudo pacman -S gst-libav

install `firefox` and `firefox-developer-edition`

https://wiki.archlinux.org/index.php/firefox

pacman -S firefox firefox-developer-edition

install python3

https://wiki.archlinux.org/index.php/Python

Setup git, github and gitlab

Need xclip

pacman -S xclip

Arch Linux on my Lenovo t470s

Sat, 30 Nov 2019 13:40:00 -0800

Install

Arch Linux is a rolling update distribution of Linux with great documentation, and a minimalist approach. I gravitate toward tools which encourage me to start small and gradually grow my understanding and skills. Arch is maybe even a little extreme for me in that regard, after installing Arch you have a system that is so bare bones it isn’t even useful. You then have all the choices in the world for how to set it up. This blog and my next one, Arch Linux First Steps, is documentation of the choices I made for my system.

This was written during my second install of Arch Linux on this laptop. I allowed my previous install to get stale for about 8 months. Arch Linux, being a rolling update distribution, was mad at me when I tried to update it after so long.

My first install was based on this helpful gist of instructions. For this second round I am following along with this youtube video as well.

WARNING This is also an install where I completely wiped my computer including the prior Windows OS. If you want a dual boot system some of these steps might still work but you’ll probably want to find another example of a dual boot Arch install.

Download ISO

I curled down the PGP key here: https://www.archlinux.org/download/ And the ISO image here: https://mirrors.ocf.berkeley.edu/archlinux/iso/2019.11.01/

gpg --keyserver-options auto-key-retrieve --verify archlinux-2019.11.01-x86_64.iso.sig

Load ISO onto USB flash drive.

diskutil unmountDisk /dev/disk2
dd if=path/to/arch.iso of=/dev/rdisk2 bs=1m

Load on Lenovo Thinkpad t470s

When the Lenovo logo shows up hit F12 to enter the “BOOT Menu”. Select the USB Drive.

Setup the wifi connection.

wifi-menu

Create Partitions

Check the disk out a bit.

lsblk

The nvme0n1 device for me has 238.5G, where nvme0n1p1 is 1G for boot and nvme0n1p2 is 237.5G. Remember, I’m documenting this during a re-install and it will likely look different for you.

Using cgdisk /dev/nvme0n1 I was shown:

1 1024.0 MiB EFI System boot
2 237.5 GiB Linux filesystem root

I want to clear these and recreate these partions in order to re-install Arch from scratch. So I navigated in cgdisk /dev/nvme0n1 to both and deleted them. I then created new partitions with the following details.

2048 -> 512M guid: ef00 name: boot EFI
all the rest guid: 8e00 (for LVM) name: Arch LVM

Create the EFI partition filesystem

mkfs.vfat -F32 -n EFI /dev/nvme0n1p1

Setup the encryption

cryptsetup -y -v luksFormat /dev/nvme0n1p2
cryptsetup open --typ luks /dev/nvme0n1p2 archlv
# check it's there
ls /dev/mapper/archlv

# create physical volume
pvcreate /dev/mapper/archlv
# create virtual group
vgcreate archvg /dev/mapper/archlv

# Under the vg, you create logical volumes.
lvcreate -L16G archvg -n swap
lvcreate -L30G archvg -n root
lvcreate -l 100%FREE archvg -n home

# Then list the block to see what you've done.
lsblk

Should have an archlv and the vgs listed under the /dev/nvme0n1p2 drive.

File systems

mkfs.ext4 /dev/mapper/archvg-root
mkfs.ext4 /dev/mapper/archvg-home
mkswap /dev/mapper/archvg-swap
swapon /dev/mapper/archvg-swap

Check that with free -m to see the swap and free memory.

Mount

These steps actually mount everything we’ve been doing in the ISO install disk onto the computers filesystem.

mount /dev/mapper/archvg-root /mnt
mkdir /mnt/boot
mount /dev/nvme0n1p1 /mnt/boot
mkdir /mnt/home
mount /dev/mapper/archvg-home /mnt/home

Install base system

Install reflector and set the mirrorlist

This tool is awesome, was happy to learn of it. My first time installing Arch I didn’t know anything about the mirrorlist and I had slow download speeds.

pacman -S reflector
reflector -c 'United States' -f 12 -l 12 --verbose --save /etc/pacman.d/mirrorlist

Use reflector --help to somewhat understand what’s happening here.

Install base arch to `/mnt`

Installs some basics into our minimal install.

pacstrap /mnt base base-devel linux linux-firmware intel-ucode sudo emacs vim tmux git networkmanager

Check it out with ls /mnt

Generate a filesystem table.

Note I learned later that I maybe should have used UUIDs in my filesystem table with the -U flag.

genfstab -p /mnt >> /mnt/etc/fstab

Check that out with cat /mnt/etc/fstab

Enter and setup the new system

Signs us out of the ISO and into our Arch install. Woot!

arch-chroot /mnt

Setup the clock.

ln -s /usr/share/zoneinfo/America/Los_Angeles /etc/localtime
hwclock --systohc --utc

Set root password with passwd.

Edit vim /etc/locale.gen to uncomment the en_US.UTF-8 UTF-8 and en_US ISO-8859-1 lines. Then run locale-gen and locale > /etc/locale.conf.

Set the hostname: echo "nackjicholson" > /etc/hostname Edit the hosts file with vim.

/etc/hosts

# Static table lookup for hostnames.
# See hosts(5) for details.
127.0.0.1 nackjicholson.localdomain nackjicholson
::1 nackjicholson.localdomain nackjicholson

Configure mkinitcpio.

Edit the “HOOKS” line in vim /etc/mkinitcpio.conf

HOOKS=(base udev autodetect modconf block keyboard encrypt lvm2 filesystems fsck)

And run the following to generate the ramdisks.

mkinitcpio -p linux

NOTE For some reason this didn’t work for me, failing with “ERROR: Hook ’lvm2’ cannot be found. I had to do pacman -S lvm2 and then try again.

bootctl --path=/boot/ install

Now edit vim /boot/loader/loader.conf

default arch
timeout 5
editor 0

Apparently that editor 0 disables a tricky hack I don’t really understand about entering the kernel. Okie doke!

Now edit vim /boot/loader/entries/arch.conf

title Arch Linux (ENCRYPTED)
linux /vmlinuz-linux
initrd /intel-ucode.img
initrd /initramfs-linux.img
options cryptdevice=UUID=XXXX:archlv root=/dev/mapper/archvg-root quiet rw

To get the value of the UUID in there, you use :read ! blkid /dev/nvme0n1p2 from within vim.

Reboot!

exit # exit arch-chroot
umount -R /mnt
reboot

Conclusion

This is a minimal install of a Arch Linux on an SSD nvme disk. I hope it all worked for you too! Feel free to email me with any corrections or issues you faced. Check out my next entry on my own personal first steps on Arch Linux, things like installing the GNOME desktop environment and setting up the WiFi and such. Thanks for reading!

The git-rebase manual review

Fri, 22 Nov 2019 21:25:00 -0800

I’ve spent most of my career being scared of git rebase. I looked on with envy as my git wizard colleagues used it like magic to rewrite history and time travel. The few times I mustered enough courage to imitate them I would rebase myself into a broken branch, a detached HEAD, and lost work. I simply gave up on rebase and decided that git merge was more my speed.

In the last six months I’ve made a concerted effort to read the manual and actually learn to use rebase from first principles. It has become an important part of my git workflow, allowing me to present cleaner pull requests and tell more compelling stories with my commits. The most useful realizations in my journey with rebase have been:

Rebase really means “move” commits.
The word “move” can also mean “redo”.
No one writes clean git logs organically.
Rebase is easier to manage when conflicts are unlikely.

Moving commits

Consider you’ve branched off master to a branch called my-new-work. When you’ve completed your work, you’d like to incorporate the changes that others have made since you diverged from master.

      A---B---C my-new-work
     /
D---E---F---G master

There are a few different ways to do this. The way I would ALWAYS do this prior to learning the semantics of rebase was to merge the changes from commit “G” into my-new-work.

git merge master

To arrive at a merge commit “X”:

      A---B---C---X my-new-work
     /           /
D---E---F-------G master

Now it is time to put in a pull request to incorporate my-new-work into master. However, pull requests lead to feedback, which lead to a new change, which leads to a new commit “H”.

      A---B---C---X---H my-new-work
     /           /
D---E---F-------G master

After I have made “H”, somebody else merged new code to master with commits “I” and “J”. I need those changes to make sure everything still works.

      A---B---C---X---H my-new-work
     /           /
D---E---F-------G---I---J master

So, I have to run git merge master my-new-work to get the code from “I” and “J”. I end up with another merge commit “Y”. Any merge conflicts which happen during this process have to be resolved before you can make commit “Y”.

      A---B---C---X---H---Y my-new-work
     /           /       /
D---E---F-------G---I---J master

Finally, my pull request is approved and I can hit the button to merge my-new-work into master; creating merge commit “Z”.

      A---B---C---X---H---Y my-new-work
     /           /       / \
D---E---F-------G---I---J---Z master

This is fine, in a way it’s simple because it accurately describes exactly what happened at every point in the time line, without any funny business or trickery. This process has worked for me for about 10 years, and I have never really been forced to give it a second thought. The way git rebase differs is that it lets you pretend you didn’t start my-new-work until after commit “J” occurred. It allows you to alter (or move) where it appears that you branched off from master. Let’s go through this set of commits again using git-rebase instead of git merge.

      A---B---C my-new-work
     /
D---E---F---G master

Before putting in the pull request, I run git rebase master.

              A---B---C my-new-work
             /
D---E---F---G master

Our colleague gives us feedback and we make commit “H” with some changes. Meanwhile, commits “I” and “J” have landed in master.

              A---B---C---H my-new-work
             /
D---E---F---G---I---J master

We can use git rebase master again to make “A” slide down the master line. The way this works in reality is that git “replays” commits “A”, “B”, “C”, and “H” one at a time as if you had started from “J”. If at any point a commit conflicts, the git rebase process will pause and exit. This is to allow you to fix the merge conflict manually. After fixing the conflict you can pickup the rebase where it left off by typing git rebase --continue. If you decide that that merge conflict is too gnarly you can decide to call the whole thing off with git rebase --abort.

                      A---B---C---H my-new-work
                     /
D---E---F---G---I---J master

Okay, let’s merge to master, creating the merge commit “Z”.

D---E---F---G---I---J---A---B---C---H---Z master

There you have it, a linear time line of what happened.

Rewriting history with `--interactive`

The majority of my time with rebase is in --interactive mode. Have you ever gotten done with something and realized you made a simple mistake like a typo? Have you ever then made a commit to fix that typo? Something with the message “typo” for instance. Then you continue reading and you discover yet another typo, and you make another commit with the message “typo”. Imagine that the “X” commits below are those kinds of meaningless commits.

  A---B---X---X---C---X my-new-work
 /
D master

An interactive rebase lets you edit, combine, or even drop commits in order to clarify the commit history of your branch. In order to perform an interactive rebase you need a place from which to start it. The commit you choose is the last commit which you want to keep intact and unchanged. In our case above “B” is a good candidate.

git rebase -i <commit hash/ref of B>

Git will then open up your editor and present you with a plan for how it’s going to apply the commits after your chosen commit. Something like:

pick bbbbbbb some meaningful commit message B
pick deadbee typo
pick fa1afe1 typo
pick ccccccc some meaningful commit message C
pick dabb1ed typo

This editor session is an opportunity to change the rebase plan by changing the word pick to another verb that rebase understands.

verb	description
pick	keep this commit in the history
edit	rebase will stop and let you edit files and commit message
break	put this on its new line and it will stop like an edit
reword	change the commit message
drop	deletes a commit and its changes
squash	fold commit into the last picked commit, concatenates messages
fixup	like squash but drops commit messages of the folded commits

With this knowledge, here is how I might apply a rebase in our branch scenario above.

reword bbbbbbb more meaningful info in message B
fixup deadbee typo
fixup fa1afe1 typo
pick ccccccc some meaningful commit message C
fixup dabb1ed typo

Here I reworded the message for commit B, and kept my typo fixes but removed the individual commits and messages for them. When I close the editor session git is going to start applying these changes.

  A---B---C my-new-work
 /
D master

Voilá! It appears that I never make mistakes and no one can ever prove otherwise!

Dangerous and magical

Seriously, go read the manual. This can be a dangerous process and you can get yourself into a mess if you’re not aware of what this command can do. The problems that the manual describes boil down to these:

Someone rebased something you depended on, thus pulling the rug from under you.
Files got renamed and you didn’t account for that.
Commits, or ranges of commits got dropped during a rebase.

My generic advice is that you should not rebase branches that you or anyone you are working with has based their work off, and you shouldn’t do anything you don’t actually understand. Git rebase does some tricky stuff, like detecting duplicate commits and auto resolving them, it can also be used to removes entire range of commits.

Consider this example from the manual. I’ve looked at it a lot and I still don’t understand what’s happening, but I know that it’s dangerous as heck and the implication is that commits disappear.

E---F---G---H---I---J  topicA

git rebase --onto topicA~5 topicA~3 topicA

Somehow this removes “F” and “G”

E---H'---I'---J'  topicA

Conclusion

There is even more to discover about this command. In this blog I covered only what I know and use in my day to day work. I’m merely an aspiring time traveler! Thanks for reading this manual review! I hope it has been helpful.

Update: The good people at https://sourcehut.org have put together the pinnacle guide on git-rebase at https://git-rebase.io/.

The git-log manual review

Sat, 05 Oct 2019 21:43:00 -0700

The git log command is for exploring the history of a git repository. You can use it to visualize, search and explore the unique story which has lead to the current state of the code. Personally, the most common ways I’ve used git log are to review my last few commits and to see the commits I’ve made since a branch diverged from master.

git log master..HEAD

This is a one-dimensional and basic usage of git log, and that has been perfectly fine for me. This is, quite frankly, all I have ever needed. Part of my journey though the git manuals is to re-discover familiar commands while looking for new ways to apply them in my workflow. I certainly learned a few things that I will be looking to employ as I see opportunities. I hope you’ll learn something from this review as well!

Basics

Visualize the whole network of changes

The log can all but replace the github network page, or fancy proprietary git GUI tools.

git log --all --graph

This is remarkably good for visualizing the basics of whose doing what, and where. I can imagine in large projects this view could get a bit unwieldy, in which case you’ll need to use it with some surgery. In normal size projects this is a good way to quickly get your bearings.

It’s worth a note that the --decorate option which shows the full ref names for every commit is enabled by default. Ignoring the reasons for why one would want to turn them off, there is the --no-decorate option which does allow for turning them off.

Hide the merge commits `--no-merges`

There are times when seeing how branches have been merged into each other is useful, but much of the time this is just noise. Filter out the noise:

git log --no-merges

Evolution of file(s)

You can easily see a list of commits which affected the given path(s).

git log ./README.md

It’s a bit trickier when a file has changed locations of course, but git log has your back with the --follow option which will follow file movements.

Tracking changes of line numbers or functions

You want to know how a certain section of a file changed.

git log -L 1,20:app.py

Show every commit, and diffs for them, which makes an alteration to the first 20 lines of a python file. Probably going to consist of changes that document every time you made new imports. :)

You can also look at commits which alter a particular function in a file.

git log -L :create_app:app.py

This will show all commits which affected the create_app function. Extremely useful!

Basic output formatting `--pretty`

The above commands automatically shows the full diffs for each commit, but if you want to see a more concise summary of each commit the commit formatting and diff options of git log are invaluable.

You can use -s, --no-patch to suppress the diff output entirely.

git log --no-patch -L :create_app:app.py

To format the commits, the --pretty options are really convenient. The options in increasingly more verbose order are oneline, short, medium, full, fuller, email, raw. You can also of course set your own formats.

git log --no-patch --pretty=oneline -L :create_app:app.py

Commit limiting

Along with being able to define ranges of commits using normal gitrevisions techniques, the git log command offers additional ways to further limit the displayed objects.

Exploring by date with `--since` and `--until`

Here’s one for the overzealous managers out there (not that you would ever read blogs about the git manual): every commit your dev team made in Q1 this year.

git log --since=2019-01-01 --until=2019-04-01 --no-merges

This shows them in reverse chronological order though, no executive is going to want to look at it like that. So we will flip them

git log --since=2019-01-01 --until=2019-04-01 --no-merges --reverse

These options are also available under the aliases --after and --before.

Exploring by `--author`

What has she ever done?

git log --author='Ada Lovelace'

Searching your log messages `--grep`

All the commits which contain the substring “separate”.

git log --grep=separate

Using some regex, let’s see only the ones that start with “separate”, but let’s also throw in the fact that we often don’t spell that word correctly.

git log --perl-regexp --regexp-ignore-case --grep='^sep[ae]rate\s'

Conclusion

I really enjoyed learning more about git log and I hope that it prompts me to ask more interesting questions of my repositories. All the best! Thanks for reading the manual review!

The gitrevisions manual review

Sat, 14 Sep 2019 15:50:00 -0700

Review

The gitrevisions manual describes all the ways you can reference objects, and ranges of objects, in git. If you have used git for a while and are comfortable using it, but don’t feel like a master of it; gitrevisions can and should be your next step. I believe this manual teaches the intermediate git user the language they need for describing the structure of their git repo. The git log --graph command shows a repository as a graph of objects, blobs(files) and trees(dirs), as it changes over time. It makes sense to me to think of revisions as coordinates in that graph. I enjoyed reading this manual, I hope you will learn something from my review of it.

Revisions

By commit with `<SHA-1>`

Each commit object has a single unique name within the repository, its SHA-1 hash e.g. 0bde21a, 0bde21a835dbb052b00895b69cf6e345a2406ed0. This hash is the ultimate identity of every revision, and can be used from anywhere. The rest of of the gitrevisions manual is about ways to target commit objects by names other than their 40-character hash, which would be difficult to work with.

Named commits with `<refname>`

It’s difficult to go far in git without using refnames. These are things like HEAD, master, or origin/master. Behind the scenes git treats these as aliases for refnames like ref/heads/master, ref/remotes/origin/master. Curious about these reference names, checkout the contents of your .git/refs directory.

There are these other special HEAD refnames which stay consistent in their meaning, even as you change or alter the state of your repository.

VARIANT	DESCRIPTION
@	An alias for HEAD, when used on its own.
HEAD	commit on which you based changes in the working tree
FETCH_HEAD	head of last fetched branch using `git fetch`
ORIG_HEAD	created internally by git when git is doing an operation that moves HEAD in a drastic way
MERGE_HEAD	commit which you into your branch with `git merge`
CHERRY_PICK_HEAD	commit you are cherry picking when you run `git cherry-pick`

Parent selection with `<rev>^[<n>]`

HEAD^ is the first parent of HEAD. Some commits, such as merge commits, have multiple parents. Using ^1 will select the first, and ^2 will select the second. What helps me is to think of ^1 as a name for “Mom” and ^2 as a name for “Dad”, and the way you express maternal-Grandma would ^1^1, parent 1 (Mom) of parent 1 (Mom). Similarly maternal Grandpa would be ^1^2. Maybe that’s a strange example but I hope it helps with understanding how the caret works in these revisions.

Generational ancestors with `<rev>~[<n>]`

HEAD~3 means go back three ancestors, and is equivalent to HEAD^^^ or HEAD^1^1^1. I find this is often a more intuitive way of targeting commits in the past than the ^ approach.

ranges

Commands like git log operate on a set of commits. When you enter a single commit to git log you’re telling it to show you the commit itself as well as any commits in its ancestry chain. Defining ranges of commits can be used to narrow and focus the information in these commands.

Exclusion

The first, most basic way to narrow a range is to specify a prefix caret, as in ^master. This means don’t show me anything which is an ancestor of master.

git log ^master HEAD

Shows the commits of your checked out branch since you diverged from master.

Dotted Ranges

The .. (two-dot) range is an alternative way of excluding commit ancestors. Running git log master..HEAD is the same as the example above using ^master HEAD.

The ... (three-dot) range defines a set of commits that are reachable from either, but not both of the commits.

git log master...HEAD

Shows what has happened on master since we diverged from it AND what happened on HEAD since we diverged from master. This is useful for showing two separate branches of changes I think.

The origin is implicitly inserted if you leave off one of the commits on either side of the range.

Simple	Implied	Meaning
..HEAD	origin..HEAD	What did I do since I diverged from the origin branch?
HEAD..	HEAD..origin	What does the origin branch have that I don’t have since I diverged from it

History by date with `<refname>@{<date>}`

This tries to get the exact commit for that reference as it was at the specified time. For example:

What was in my local master branch 1 week ago?

git show 'master@{1 week ago}'=

Shortcut to the upstream branch with `<branchname>@{upstream}`

Using @{upstream} will reference the commit of the branch configured by branch.<name>.remote and branch.<name>.merge

Shortcut to the push branch with `<branchname>@{push}`

Using @{push} will reference the commit of the branch configured by push.default. The manual describes this as a reference to “where we would push to if git push were run while branchname was checked out.

Refer to a file or subtree of a revision with `<rev>:<path>`

Targets files or a directory at the given path. For instance:

git show origin/master:README.md

git diff origin/master:blog.org

To show the difference between the blog.org file in the working tree, and the one on origin/master. That first argument origin/master is a revision. Git diff can take a path revision to show a diff between blog.org in my working tree, and the repositories README.md on origin. Why do this? I have no idea, but it works!

git diff origin/master:./README.md blog.org=

Now, throwing it all together! What’s the diff between origin master blog.org, and origin master blog.org from 1 week ago?

git diff origin/master:blog.org 'origin/master@{1 week ago}:blog.org'

Conclusion

After reading and reviewing this manual, I am planning to commands which leverage the power of revisions like git-log, git-rebase, and git-push. I think I’ll be coming back to this document pretty often as I dig into these other tools. As always, thank you so much for reading this manual review!

The giteveryday manual review

Sun, 08 Sep 2019 17:09:00 -0700

The giteveryday(7) manual seeks to describe a minimal set of useful commands for everyday usage of git.

There really isn’t much of a reason to read this manual. I actually found it rather interesting, but only from an archaeology of software development point of view. It feels a bit outdated for a beginning programmer. It describes ways to email patches to developers, and apply patches from emails, as well as how to spin up a git daemon or a git web front-end. I recommend scanning this manual quickly for the example commands that are being described and only reading more if something catches your interest.

Thanks for reading this manual review!

The gittutorial manual review

Mon, 02 Sep 2019 22:28:00 -0700

The git tutorial manual is actually two separate manuals, gittutorial(7) and gittutorial-2(7). I will be reviewing them both in this post, followed by some brief summaries of what I learned or thought was noteworthy while reading them.

Review

I think if I were an absolute beginner and I started with these tutorials, the gittutorial would be slightly helpful, and the gittutorial-2 would be confusing. I don’t think the manual is where true beginners typically start with git. I would guess many learn the basics of git as part of the learning the basics of programming. The gittutorial manual is a reiteration of the basics. It gives enough context on git to start using it, though I wouldn’t say it does so particularly well. There are probably better and more practical getting started tutorials available from a quick google search. This manual feels outdated because the examples are of Alice and Bob sharing a single filesystem where they both have user directories and their own repositories. That’s not typically how we work with git these days, and I don’t think it’s a helpful distraction for todays beginner programmer.

In contrast, the gittutorial-2 manual quickly gets you straight into the internal architecture of git in a way that could be overwhelming for a beginner. As a developer who has been using git for 10 years without ever reading the manual though, this was illuminating! This felt like information I had been missing. While it may be overwhelming for a beginner perhaps it is best to be forced to go a little deeper in order to build a foundational and lasting understanding. I think I would recommend a beginner to read gittutorial-2 in their first few months of learning, but not right away. This manual aims to teach how to think about a git repository as a system and database of information. I think that vantage can potentially be a leg-up in learning to wield git beyond the basics.

gittutorial

Some things I learned and plan to use.

log and .. ranges

git log --stat --summary

This is a nice view on the log.

git fetch /home/bob/myrepo master
git log -p HEAD..FETCH_HEAD

The usage of git fetch and the FETCH_HEAD symbol as a way to peek at code. This also demonstrates the concept of commit ranges, which I rarely use and probably could use more.

The two-dot .. form means: show everything that is reachable from FETCH_HEAD, but exclude anything that is reachable from HEAD. The example in the manual is a peek at Bob’s changes absent of Alice’s changes.

The three-dot ... form means: show everything that is reachable from either, but exclude anything that is reachable from both. This is a view of Alice’s changes combined with Bob’s changes.

git log --since="2 weeks ago" # commits from the last 2 week

Could be handy occasionally.

gitk

I didn’t know this existed. Its a Tkinter based UI for git! It’s pretty old-school cool. I’m happy that it exists and I’m going to find reasons to open it up in strange situations when pairing with colleagues, just to see their reactions.

git show

An introduction to seeing the details of commits. I think I often go straight to github when I need to find a commit, when it may be much easier to use git show.

Git ancestry targeting:

HEAD^, HEAD^1 is the parent of HEAD
HEAD^^, HEAD^2 is the grandparent of HEAD
HEAD^4 is the great-great grandparent of HEAD

git grep

Search a commit for a pattern. I did not know this existed.

gittutorial-2

Objects

Git “commit” objects refer to snapshots of the whole directory structure in a repo known as a “tree”. Each directory is a tree object in git, and file data is stored in “blobs”. The trees can themselves refer to other trees (sub directories). Using git cat-file and git ls-tree you can introspect the objects stored by git.

The index file

The index let’s git show us the difference between the state of the working directory and the last commit. By default git uses the index to create commits, not the working directory. What I learned here, which I already knew by using git was I can use git add to manipulate the index in order to shape commits, without necessarily discarding my working tree. In practice I don’t really do this often, I just add everything to a commit.

Conclusion

The gittutorial-2 is worth a look, you’ll learn about how git works in a way which should help you master it. There’s still a lot more for me to read about git. My plan is to move on to the giteveryday manual. I’ve also been trying to work my way through the Git User’s Manual, as well as the manual for Magit; a great interface to git from within Emacs. It doesn’t shield you from git, but like gitk, it gives you a different way to visualize and manipulate the database of git objects.

Go RTFM! And as always, thanks for reading the manual review!

The git manual review

Mon, 02 Sep 2019 21:04:00 -0700

Running man git opens up the entrypoint on git documentation, there are separate manuals for every command, references to the Git User’s Manual and the gittutorial, as well as giteveryday. I’m not going to be able to do a single write-up on all this information, it’s going to take me like a year to read it all! So I think what I’ll do with my git review is spread it out over multiple blogs a little bit at a time. I’ll start with gittutorial(7) soon.

As I read and review git manuals, I will post links to them from here. This is now the table of contents for my git manual reviews.

Remember to RTFM. Thanks for reading!

Reviews:

The curl manual review

Mon, 19 Aug 2019 23:22:00 -0700

Review

I can now recommend the curl manual. It was definitely worth the read. I discovered things about curl that I didn’t know before and a great deal about how to more effectively use it. The manual contains a lot of information about niche topics like socks, proxies, and FTP servers. Most of which I skimmed. I tried to concentrate on aspects of curl I thought I would use often.

I chose to read it because I wanted to better understand the capabilities of curl and to recognize common spots where employing it could save me time and effort. I found the URL syntax documentation especially informative.

Because I work for a company focused on the climate, many of the examples below involve this index of global hourly NOAA weather data. It is organized by years, and USAF and WBAN station ids. Referring to the structure of this directory may help you better understand the examples below.

https://www.ncei.noaa.gov/data/global-hourly/

Basic `curl`

Output content to stdout

The station id for SFO (San Francisco International Airport) is 72494023234. This request prints the csv data to stdout.

curl https://www.ncei.noaa.gov/data/global-hourly/access/2016/72494023234.csv

Using grep on the output to view data for a particular day.

curl \
  https://www.ncei.noaa.gov/data/global-hourly/access/2016/72494023234.csv \
  | grep '2016-07-28'

Downloading files with `-o`, `--output` and `-O`, `--remote-name`

Downloading the SFO data.

mkdir -p scratch/2016/
curl \
  https://www.ncei.noaa.gov/data/global-hourly/access/2016/72494023234.csv \
  --output scratch/2016/sf-weather.csv

Using the alias -o instead.

curl \
  https://www.ncei.noaa.gov/data/global-hourly/access/2016/72494023234.csv \
  -o scratch/2016/sf-weather.csv

Using a progress bar with `-#`, `--progress-bar`

curl will display a progress meter with the amount of transferred information and how long the request took.

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 6825k  100 6825k    0     0  1988k      0  0:00:03  0:00:03 --:--:-- 1988k

Providing -# or --progress-bar arguments will display a progress bar of the transfer instead.

curl \
  https://www.ncei.noaa.gov/data/global-hourly/access/2016/72494023234.csv \
  -o scratch/2016/sf-weather.csv \
  -#
############################################################################### 100.0%

If you don’t want any progress meter, use -s, --silent.

API requests with `--request`, `--data`, and `--header`

The NOAA website doesn’t provide any use for POST requests, so consider a micro-blogging API which is completely made up. ;-)

curl \
    -X POST \
    --url 'https://api.microblog.com/1.1/statuses/update.json' \
    -d '{"status": "hello"}' \
    -H 'Content-Type: application/json' \
    -H 'authorization: OAuth ...oauth info...'

Args in this request:

-X, --request argument varies the HTTP verb of the request.
--url is the ordinary positional URL argument as a named argument.
-d, --data is the data to post. In this case a JSON document.
-H, --header supplies HTTP Headers to the post.

To post data contained in a file, use the @ symbol with the --data arg:

--data @path/to/big_data_file.json

Making HEAD requests with `-I`, `--head`

Retrieve only the header of a document. Useful when the content of a request is not consequential. I use this sometimes as a simple test of network connectivity to a server, or to see the raw 301 response before a redirected request.

curl \
  -I \
  https://www.ncei.noaa.gov/data/global-hourly/access/2016/72494023234.csv

Produces:

HTTP/1.1 200 OK
Date: Mon, 26 Aug 2019 04:37:49 GMT
Server: Apache
Strict-Transport-Security: max-age=31536000
Last-Modified: Wed, 19 Dec 2018 17:25:32 GMT
ETag: "ae57fe0-6aa4e2-57d634d14db00"
Accept-Ranges: bytes
Content-Length: 6989026
Content-Type: text/csv
Access-Control-Allow-Origin: *
Access-Control-Allow-Headers: X-Requested-With, Content-Type
Connection: close

Following redirects with `-L`, `--location`

Web pages can be tricky, hitting one might move you to another. Consider the following request to https://google.com.

curl https://google.com

<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="https://www.google.com/">here</A>.
</BODY></HTML>

This doesn’t produce the full google web page; it is a redirect to the full google experience. Looking into the response headers we can see the details of this redirect.

curl -I https://google.com

HTTP/2 301
location: https://www.google.com/
content-type: text/html; charset=UTF-8
date: Mon, 26 Aug 2019 04:46:00 GMT
expires: Wed, 25 Sep 2019 04:46:00 GMT
cache-control: public, max-age=2592000
server: gws
content-length: 220
x-xss-protection: 0
x-frame-options: SAMEORIGIN
alt-svc: quic=":443"; ma=2592000; v="46,43,39"

By default curl isn’t following this redirect to the google home page in the same way your web browser does. The -L, --location argument makes the request follow any redirects through to the final destination.

curl -L https://google.com

The response here is a much more complicated HTML page for the google website.

curl -I -L https://google.com

This will show the header responses for both the redirect and the final destination.

Essential `curl`

Downloading files with remote information.

The -O, --remote-name option lets us use file names from the server. Similarly, -R, --remote-time attempts to use the servers timestamp of the file if it is available.

Uses the remote name of the file and saves the file to the current working directory.

curl -OR https://www.ncei.noaa.gov/data/global-hourly/access/2016/72494023234.csv

The file is downloaded to ./72494023234.csv, and ls -l reveals that even though I am downloading this data in 2019, the last modified time of this file was in 2018.

-rw-r--r--  1 nackjicholson  staff  6989026 Dec 19  2018 72494023234.csv

Storing request header information with `-D`, `--dump-header`

Sometimes when inspecting requests or API responses, you may want to dump header information along with the request.

curl -OR \
  -D headers.txt \
  https://www.ncei.noaa.gov/data/global-hourly/access/2016/72494023234.csv

Sending form data with `-F`, `--form`

Just do man curl and search for --form, and read about it. It’s going to be more informative than anything I can say here.

Features:

Send files with @
Send input fields
Specify content types per field
Load content from text fields with <

Ignore SSL with `--insecure`

When hitting routes which require SSL curl will fail.

curl 'https://xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-xxxxxxxxx.us-west-2.elb.amazonaws.com'
curl: (51) SSL: no alternative certificate subject name matches target host name 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-xxxxxxxxx.us-west-2.elb.amazonaws.com'

I encounter this occassionally trying to connect to an AWS load balancer before associating them with a DNS name and a valid SSL cert.

curl --insecure 'https://xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-xxxxxxxxx.us-west-2.elb.amazonaws.com'

You don’t know `curl`

Sending multiple requests using URL Syntax

The url sent to curl can be given multiple values to create multiple requests. Using syntax {foo,bar} brackets creates sets which replace parts of a URL. For example, we can request data for the New York La Guardia Airport (72503014732) and San Francisco International Airport (72494023234) in a single request.

curl \
    'https://www.ncei.noaa.gov/data/global-hourly/access/2016/{7294023234,72503014732}.csv' \
    -o scratch/sf-weather.csv \
    -o scratch/ny-weather.csv

Using two -o arguments writes two output files, one for each request, in the order they are defined. We can also use [..] to create ranges of values. The NOAA data is organized into directories by year. We can create a URL that will get us the SFO and LGA data for a sequence of years. We could then use as many --output arguments as necessary to write all the files out, but there is an easier way by templating the --output names as well.

curl \
    'https://www.ncei.noaa.gov/data/global-hourly/access/[2012-2019]/{7294023234,72 503014732}.csv' \
    --create-dirs \
    --progress-bar \
    --output 'scratch/#1/#2-weather.csv'

This will download 16 files for SFO and LGA, creating a directory structure using --create-dirs and the names #1 and #2 which correspond respectively to the two substitution parts of our URL in the order they are defined. The first is the range of years, and the second is the set of station ids.

scratch
├── 2012
│   ├── 72503014732-weather.csv
│   └── 7294023234-weather.csv
├── 2013
│   ├── 72503014732-weather.csv
│   └── 7294023234-weather.csv
├── 2014
│   ├── 72503014732-weather.csv
│   └── 7294023234-weather.csv
├── 2015
│   ├── 72503014732-weather.csv
│   └── 7294023234-weather.csv
├── 2016
│   ├── 72503014732-weather.csv
│   └── 7294023234-weather.csv
├── 2017
│   ├── 72503014732-weather.csv
│   └── 7294023234-weather.csv
├── 2018
│   ├── 72503014732-weather.csv
│   └── 7294023234-weather.csv
└── 2019
    ├── 72503014732-weather.csv
    └── 7294023234-weather.csv

8 directories, 16 files

Saving request configurations with `-K`, `--config`

When requests start getting large, difficult to remember, or tedious to type; you can use the --config feature of curl to save the request options to a file.

This saves our complicated configuration.

bulk_station_download.txt

url = "https://www.ncei.noaa.gov/data/global-hourly/access/[2012-2019]/{7294023234,72503014732}.csv"
output = "scratch/#1/#2-weather.csv"
--create-dirs
--progress-bar

Now we can perform this request using:

curl -K bulk_station_download.txt

Separate Requests with `-:`, `--next`

This will let you chain posts in order. A sample use for this might be to do a POST creating a resource, and then follow it with GET to make sure it’s accessible.

curl \
  -X POST \
  -H 'Content-Type: application/json' \
  -d @my_thing.json \
  http://api.example.com/things \
  -: \
  http://api.example.com/things/my_thing \
  -o my_thing.json

Writing out request information and statistics with `-w`, `--write-out`

This StackOverflow has great information about how to do this, and the curl manual has a list of the templating variables you can use in your -w template files.

curl_request_info.txt

   time_namelookup:  %{time_namelookup}\n
      time_connect:  %{time_connect}\n
   time_appconnect:  %{time_appconnect}\n
  time_pretransfer:  %{time_pretransfer}\n
     time_redirect:  %{time_redirect}\n
time_starttransfer:  %{time_starttransfer}\n
                   ----------\n
        time_total:  %{time_total}\n

Then make a request. This one doesn’t even download the file, it just pipes it’s content to /dev/null and shows the stats.

curl -w "@curl_request_info.txt" -s -o /dev/null https://www.ncei.noaa.gov/data/global-hourly/access/2016/72494023234.csv

And you’ll get back something like:

   time_namelookup:  0.001
      time_connect:  0.037
   time_appconnect:  0.000
  time_pretransfer:  0.037
     time_redirect:  0.000
time_starttransfer:  0.092
                   ----------
        time_total:  0.164

Conclusion

There were a few other things I thought looked cool, but didn’t play with.

Abort slow downloads -Y, --speed-limit in bytes per seconds
Abort slow downloads after a given time -y, --speed-time
Retries with --retry

Remember to RTFM! Thanks for reading my review.

William Vaughn - San Francisco, CA

Mon, 01 Jan 0001 00:00:00 +0000

Software Engineering Leader - will@willvaughn.org

Objective

To build, architect, and operate data-centric software applications. To create productive software environments that challenge, teach, grow, and support professional crafts-persons.

Employment

Carbon Lighthouse - San Francisco, CA

2016-Present – Principal SWE < VP of Software < SWE Manager < Lead SWE < Sr. SWE
Company founded to stop climate change through profitable decarbonization of commercial real estate.
I started as an individual contributor, and progressed to organizational leader overseeing a cross-functional team building energy modeling applications on AWS, Kubernetes, and Python.

Designed and built data ingest system, foundational to collecting and analyzing hundreds of millions of timeseries points from building sensors.
Led company’s data-driven transition by implementing and evangelizing a modern data and reporting stack on Snowflake, Fivetran, and dbt.
Reduced operational overhead by leading migration from Apache HBASE to AWS DynamoDb.
Turned local team knowledge into company-wide knowledge by creating tooling and infrastructure to publish documentation from dozens of code repositories to a single internal site.
Increased learning pace of a product organization delivering new features once a quarter, to shipping multiple times per week.
Guided direct reports through COVID-19 remote transitions, two rounds of layoffs, and a business pivot. Forged a gritty team through a difficult downturn.
Revamped software engineer hiring process to remote positions and a higher bar.

R/GA - Portland OR

2015-2016 – Sr. SWE
Premier design, branding, and advertising agency.
Was embedded at Nike Headquarters on the Nike By You (aka Nike iD) product customization team.

Delivered mobile frontend for one of Nike’s most profitable online experiences.
Unlocked new sales drivers by working hand-in-hand with Nike.com teams to provide API integration between our embedded app and product pages.

Cascade Energy - Portland, OR

2012-2015 – Sr. SWE < SWE
Industrial Energy Efficiency Consultancy.
Full-stack development of industrial energy management tool SENSEI.

Built scalable utility meter ingest system on AWS EC2, S3, SQS, and DynamoDb.
Developed calculation UI and API enabling expert mechanical engineers to create models on time-series measurements.

ID Branding/Liquid Agency - Portland, OR

2010-2012 – Jr. SWE
Brand advertising, design, strategy and experience agency. ID Branding was acquired by Liquid Agency in early 2011.

High level design and brand communication web applications for clients.
Microsites for Nike, Trimet, and other regional clients.

Education

Portland State University, September 2009-Jan 2010 (No Degree)
Pacific University, August 2004 – March 2006 (No Degree)
University of Oregon September 2002 – June 2004 (No Degree)
La Grande High School, Graduated 2002

William Vaughn

Psalm 106

Psalm 52

Leviticus 16:20-22, 17:10-16

Genesis

Job 38:31-33; Job 39:9

Job 21:7; Job 22:17; Job 23:12-14

Genesis 11:4-6

Can I read the whole Bible this year?

Emacs shell-command

VP of Emacs: The Job

People, Process, System, Strategy

People

Process

System

Strategy

Managerial Economics

Hiring & Recruitment

How Emacs Helps

More in this series

VP of Emacs: Meeting Notes

Meeting notes with Org-roam dailies

Navigating to daily notes

Quick capture to daily notes

Meeting capture templates

Review and grooming

More in this series

VP of Emacs: A Personal Wiki In Plain Text

A personal wiki with Org and Org-roam

Everyone and every thing gets their own org-roam note

One on one meetings

Coming up next

Moving to sourcehut

PyPA Python Packaging User Guide Review

Highlights

IRC Services at Home with k3s

/var/lib/rancher/k3s/server/manifests/traefik.yaml

traefik-ingress/base/routes.yaml

bitlbee/base/service.yaml

bitlbee/base/pvc-bitlbee.yaml

bitlbee/base/pvc-signald.yaml

bitlbee/base/deployment.yaml

znc/base/service.yaml

znc/base/pvc.yaml

znc/base/deployment.yaml

ZNC Webadmin Listeners

Running Bitlbee & Signald with Docker

Overview

Setup Storage & Cross Communication Volumes

Run the Containers:

Login & Verify Device with Signal

Create a Systemd User Unit

Conclusion

Running ZNC Bouncer With K3s

Implementation

Overview

Persistent Volume for ZNC Config

ZNC Deployment

ZNC NodePort Service

Connecting from ERC

Conclusion

My Personal IRC Renaissance

Freenode->Libera + System Crafters

ZNC

0x0.st

Bitlbee & Signald

crafter-bot

Conclusion

The git-add manual review

Basics

Pick and Choose with --edit

Interactive additions with -p, --patch

Conclusion

Arch Linux on my Toshiba Chromebook 2 (2014)

Specs of the laptop

Prepping the Chromebook to be hacked on

Installation process

Partitioning

Mounting the new filesystem

Mirror List

Everyone and every thing gets their own `org-roam` note

Pick and Choose with `--edit`

Interactive additions with `-p`, `--patch`

Formatting the disk with `parted`

Archiving `DONE` items

Install `zsh`

Install `gnome` and `gnome-extra`

install `firefox` and `firefox-developer-edition`