Using the Stable Diffusion WebUi

Using the Stable Diffusion WebUi

Installing Stable Diffusion is one thing using the Stable Diffusion WebUI to create decent checkpoints, and understanding the different weights and measures, sampling sets, and prompts is where the real magic (and frustration) happen.

Why Stable Diffusion?

Recently I wrote an article for my employer allcode.com on how to set up Stable Diffusion and use Dreambooth to train models based on your own images to personalise the outputs, thus allowing you to create artistic renditions of yourself with various prompts.

The critical element is that it can run on a local gaming PC (although you do have to train the models on a pay-on-demand server). I highly recommend you read that article if you are interested in how to set up the various systems.

A long, long time ago, I used to love art; painting, drawing, and sculpting were really enjoyable for me. But sadly, over the years, I’ve let the meagre skills I had atrophy and now trying to return to square one is just disheartening.

So I’m finding this fusion between coding, AI, writing and art really fascinating. I see a lot of artists complaining about AI art (and with good reason). I’m definitely finding people selling AI art really dubious and troubling. That said, it’s here, it’s not going away, and as I’ll show throughout these blog posts, it is not so easy. I can see it being a great tool for artists and a problem in years to come.

 

AI generated image of me as legolas

What’s the plan

Over a series of posts, I’m going to write up what all the parts of the Stable Diffusion WebUI interface do, what they mean and how they relate back to things like Dreambooth and training your own models. I’ll include a little about configs for training models and probably show how changes to the training affected the outputs. 

 

What won’t be covered

I’m not going to explain all the various installs, libraries, variations and forks of Stable Diffusion, nor am I going to compare the outputs to other AI art creators like Midsummer etc. I’m also not going to answer questions on how to resolve code breakages that happen, unless I happen to experience one while writing.  

Maybe when I’ve finished playing with Stable Diffusion, I’ll look at others in the future.

 

jamestyrrell as a Warhammer 40k Space Marine SD, Dreambooth

The WebUI homepage

Today I’ve set the scene for what I hope to create so for now I’m just going to give a high overview of the various tabs and what they do.

Holy sh*t, there are a lot of options here

If you’ve followed my tutorial at allcode.com, you’ve already installed Stable Diffusion, run your personalised images through Dreambooth and are ready to get down to business. The first thing you see when you open the WebUI is a lot of options that aside from the prompt option will make little sense. Not to even mention that this is tab 1 of 8.

WebUI screen shot

txt2img

This is probably the most recognizable tab, as it is the key point of popularity for the Stable Diffusion craze. It’s where you type in prompts (both negative and positive) and generate images. On a default install of Stable Diffusion you can still get access to the massive wealth of information on artists, celebrities and politicians when creating these prompts. However, do remember to complete the Dreambooth training if you want to personalise any generated images. 

img2img

Image to image AI art generation (img2img) uses the same principle as that of txt2img above. Users still enter in prompts for the AI. The main difference between these two is that a base image is included into the scenario

Users upload a base photo where the AI applies changes based on entered prompts. Continuous iteration of this process creates refined and sophisticated art.

Extras

In this section you have the ability to upscale your produced images to bigger sizes, all sizes are usually produced as 512×512 and other tools to improve the on images that you like but feel need work

  • GFPGAN, neural network that fixes faces
  • CodeFormer, face restoration tool as an alternative to GFPGAN
  • RealESRGAN, neural network upscaler
  • ESRGAN, neural network upscaler with a lot of third party models
  • SwinIR and Swin2SR neural network upscalers
  • LDSR, Latent diffusion super resolution upscaling

PNG info

If you have an image generated by Simple-Diffusion it should includes all the details of the prompt, negative prompts, seed, model and other information used to generate it. So rather than trying to rememver a prompt that you liked you can find the produced image and drag-and-drop the image onto the PNG info tab you will recover all the info and then send it to other areas of the WebUI

Checkpoint Merger

Combine and blend different checkpoints and models. 

Train

If you have a monster of a machine with 24Gb of VRAM on your GPU you can train your models locally, normally trying will just make the system crash and laugh at you. If you want you there is a “Dreambooth extension” which can be found in the Extensions tab described below which can help lower powered systems train but otherwise you’re better off training models on a service like https://www.runpod.io/ until either the systems get smarter (less GPU required) or the graphics cards get cheaper.  

Settings

The settings tab for Stable Diffusion WebUI allows you to customize the various options and preferences. The options and preferences available include things like the color scheme, where output images are stored, and the way that the program interacts with other programs or devices. In general, the settings tab is where you can fine-tune the way that the Stable Diffusion WebUI works to suit your needs and preferences.

Extensions

An extension is a small software program that can be installed in the Stable-Diffusion WebUI to add or modify the functionality. Extensions can be used to add features to the WebUI, such as a new tab, or to modify the behavior of Stable Diffusion, such as reducing the load for Dreambooth or blocking certain websites. Extensions can be installed and updated from this tab, and once installed, they can be turned on or off as required.

Stable Diffusion v2.0 and Prompts

Stable Diffusion v2.0 and Prompts

Stable Diffusion v2.0 and prompts

The update brings improvements to SD’s text-to-image diffusion models, includes a powerful image upscaler, updates its Inpainting model, and more.

Also it appears to have made massive updates to the prompting system comparing Stable diffusion v1.4 to v2 gives pretty vast differences. They produced a prompts guide which was online but seems to have disappeared?

But irritatingly it can’t be downloaded as a .pdf, me being a massive nerd said f**k that and used python to pull the svgs and created a pdf.

Here are a few examples of how different things can get

So here are a few examples with the relevant seeds and prompts to show you how dramatically things have changed.   Also, it should be mentioned a lot of celebrities have been removed from the model, NSFW images and famous artists like Rutkowski have been purged due to complaints of their style being essentially stolen by the machine.

What has been removed from Stable Diffusion’s training data, though, is nude and pornographic images. AI image generators are already being used to generate NSFW output, including both photorealistic and anime-style pictures. However, these models can also be used to generate NSFW imagery resembling specific individuals (known as non-consensual pornography) and images of child abuse.

Of course, there are a lot of angry Incels now wailing about censorship. This is bullshit, this stuff is open-source. No doubt a horde of horny teenagers are using PornHub and other sites to train NSFW models. 

Prompt: Gandalf, d & d, fantasy, intricate, elegant, highly detailed, digital painting, artstation, concept art, matte, sharp focus, illustration, hearthstone, art by artgerm and greg rutkowski and alphonse mucha
Negative prompt: cartoon, 3d, ugly face, (disfigured), (bad art), (deformed), (poorly drawn), (extra limbs), strange colours, blurry, boring, sketch, lacklustre, repetitive, cropped, hands

Steps: 60, Sampler: DDIM, CFG scale: 12, Seed: 1940895508, Face restoration: GFPGAN, Size: 512×512, Model hash: 7460a6fa, Batch size: 6, Batch pos: 0

gandalf, d & d, fantasy, intricate, elegant, highly detailed, digital painting, artstation, concept art, matte, sharp focus, illustration, hearthstone, art by artgerm and greg rutkowski and alphonse mucha<br />
Negative prompt: cartoon, 3d, ugly face, (disfigured), (bad art), (deformed), (poorly drawn), (extra limbs), strange colours, blurry, boring, sketch, lacklustre, repetitive, cropped, hands<br />
Steps: 60, Sampler: DDIM, CFG scale: 12, Seed: 1940895508, Face restoration: GFPGAN, Size: 512x512, Model hash: 7460a6fa, Batch size: 6, Batch pos: 0

Stable Diffusion 1.4

gandalf, d & d, fantasy, intricate, elegant, highly detailed, digital painting, artstation, concept art, matte, sharp focus, illustration, hearthstone, art by artgerm and greg rutkowski and alphonse mucha<br />
Negative prompt: cartoon, 3d, ugly face, (disfigured), (bad art), (deformed), (poorly drawn), (extra limbs), strange colours, blurry, boring, sketch, lacklustre, repetitive, cropped, hands<br />
Steps: 60, Sampler: DDIM, CFG scale: 12, Seed: 1940895508, Face restoration: GFPGAN, Size: 512x512, Model hash: 7460a6fa, Batch size: 6, Batch pos: 0

Stable Diffusion 2.0

Prompt: Portrait digital art of Bill Murray from Scrooged (Arcane). wearing a suit, Christmas,

Negative prompt: cartoon, 3d, ugly face, (disfigured), (bad art), (deformed), (poorly drawn), (extra limbs), strange colours, blurry, boring, sketch, lacklustre, repetitive, cropped, hands

Steps: 40, Sampler: DDIM, CFG scale: 13, Seed: 3408805356, Face restoration: GFPGAN, Size: 512×512, Model hash: 7460a6fa, Batch size: 6, Batch pos: 0

Portrait digital art of Bill Murray from Scrooged (Arcane). wearing a suit, Christmas,

Stable Diffusion 1.4

Portrait digital art of Bill Murray from Scrooged (Arcane). wearing a suit, Christmas,<br />
Negative prompt: cartoon, 3d, ugly face, (disfigured), (bad art), (deformed), (poorly drawn), (extra limbs), strange colours, blurry, boring, sketch, lacklustre, repetitive, cropped, hands<br />
Steps: 40, Sampler: DDIM, CFG scale: 13, Seed: 3408805360, Face restoration: GFPGAN, Size: 512x512, Model hash: 09dd2ae4, Batch size: 6, Batch pos: 4

Stable Diffusion 2.0

What’s next

I’ve started writing up how the Stable-Diffusion WebUI works and will probably wite up how to scrape a google SVG slideshow.  

AI Generated Comics and Copyright

AI Generated Comics and Copyright

What is the full story

So recently the writer Kris Kashtanova used Midjournery AI to generate all the images in a comic, and then just recently the U.S. Copyright Office appears to be backtracking on its decision to grant protection to an AI-generated comic book. now on the face of it this seems like a no-brainer of course he shouldn’t be able to copyright the images generated by a machine, no matter how carefully they were obliged to type the prompts in to get the images they desired for the story (no mean feat). 

However, there is a contentious element here for me, it looks like the USPTO is removing the copyright from the entire book including the writing. Now perhaps this isn’t the case and I’m misreading things but even in a comic book art and writing are two separate elements. Kashtanova didn’t use ChatGPT to write the story or text in the comic, that was his own work, the book should still be copyrighted to him even if the art isn’t his. 

It would be like me finding royalty free for use images online and then using them to illustrate my story, am I now in danger of losing my other rights, what about a talented painter who uses ChatGPT to help write a story with prompts and then paints the scenes, what happens to their rights? 

The future is going to get even messier

Sometimes it is hard to realize the incredible progress we’ve made in terms of cheap, accessible computing power and AI. Then something like this comes along and people are shocked, but this is really just the beginning of this, what we are witnessing is the tip of the AI glacier that is about crash down on society

In 10 years AI art generators will be an order of magnitude better, and our ability to prompt them with far more nuanced, faces, eyes, and body proportions are all hit or miss currently. In 10 or 20 years we’ll see fusions of deepfakes, art and media that will change a lot of industries, forever. 

Let’s take an example I love the film Scrooged so in 20 years I think hey let’s watch that, but you know I’ll make it a cartoon, like Arcane and add labelling to the action scenes like in Scott Pilgrim vs the World, maybe change out the bad guy for Alan Rickman. Then maybe 20/30 minutes later, I watch my new personalised version of Scrooged. 

That’s not hyperbole, that’s just where things are going to be on the route we’re currently heading. If you think that sounds crazy, remember that 20 years ago the best kind of phone you could have was a Nokia. 

 

Portrait digital art of Bill Murray from Scrooged (Arcane). wearing a suit, Christmas,

What’s next

I’m going to write more about this because it really interests me and as I study and learn about it I’ll keep a running commentary on my blog.

It’s the first time in a long time I’ve seen a subject that’s blown me away like this so hopefully, that will translate into me getting back into writing fiction, I’ll just have to be careful how I illustrate it I guess ;). 

Daily writing – What can happen in a second?

Daily writing – What can happen in a second?

So I haven’t written in forever, daily writing was my intent until I got a new job, moved apartment, started working full time, also have started exercising. I Have no routine and the last big task to write, based on the masterclass by Neil Gaiman, was unwieldy and while I reached the end I need to revise and rework it.

To break the dry spell I’m going to try and use the book 642 things to write about. It gives you a topic to write just a short paragraph a day on. The hope is if I can just get into the habit of sitting down and writing every day for a few minutes, I can develop that into a habit of writing what I love, science fiction and fantasy. I’ll also try to include a photo I take each day.

What can happen in a second?

A lot of things can happen in a second, you can fall in love, you can make a life-changing decision, step into the path of a vehicle. Anything and everything in our lives can be traced back to a choice we have made.

That’s quite a lot of weight to put on our shoulders though, also it doesn’t mean that other people’s choices have no impact on our lives, the actions that you have no control over, the random vagueries of life that hit you on some idle Tuesday.

Or in the case of my office getting up for one second and having my cat Ninja take his rightful place on my office chair. He seems happy enough with that choice.

Ninja the chonk, the real reason I get nothing done.

Character Situation Exercise

Character Situation Exercise

Author’s note: Following the Neil Gaiman Master Class series, he offers the following exercise: Using the questions below, create a character. This person can be based on people you know or have completely invented. Write a sentence or two describing what the character wants.

  • What is their name? Age? Gender?
  • What do they look like?
  • What’s in their purse or pockets?
  • What’s their favourite thing to do?
  • What do they hate?
  • What matters to them more than anything in the world?

The Questions

What is the character’s name? Age? Gender?

Pippa, ??, Female

What does the character look like?

Brunette (possibly), but she’s dyed her hair so many colours she doesn’t remember the original anymore, tattoos adorn her entire body, but despite a lithe and attractive compact form she normally wears relaxed clothes, jeans, and loose t-shirts to hide it.

Her face is habitually fixed in a wide grin, eyes somewhere between hazel and amber, always twinkling with a mischievous light. Some light make-up on her eyes, maybe lipstick if she’s dressing up or trying to impress.

Both her ears have at least two piercings while her tongue has a single metal bar in through it.

What’s in the character’s purse or pockets?

Probably her wallet, keys, and phone at a minimum. Also, a pen or 3, a notepad, a lighter, coins distributed through pockets, and at least 1 condom.

What’s the character’s favorite thing to do?

Chilling with friends, making people laugh, and being around other like-minded people (sometimes all at the same moment).

What does the character hate?

Liars, bigots, and hypocrites. Also being ignored or shunned.

What matters to the character more than anything in the world?

Her friends and the people she loves. She would do anything for them and wants them to know that they can rely on her to be there for them.

====

Writing Task

Using the character you created above, write a conversation or a situation where this person cannot get what they want. You may find other characters emerge from this—let it happen. Try opening the scene at the main point of tension—for instance, in the middle of a fight— and fill in the necessary details as the scene plays out.

A Beginning

Pippa practically bounced through the opening, feeling her heart skip a beat as the heavy pub door thumped shut behind her. The packed bar assailed all her senses, the frigid air outside replaced by the blast of voices, loud music, heat, stale smell of sweat, and smoke. Pausing on the threshold, her breath caught in her throat as a room of eyes turned to stare, judge, and undress her.

Pushing up on her tiptoes, she tried to survey the room. Not an easy task, despite wearing her boots, she was still a full head shorter than most of the crowd. Cigarette smoke hung like a mist around the tables, causing her eyes to prickle. Still, she felt a fractional easing in her shoulders when she saw the room was full of familiar faces. This was her bar, and these were her people.

The table across from the entrance caught her eye, and hands, beers, and bottles were raised in greeting. “Pippa!” came a chorus of voices, and numerous hands beckoned her over. She raised her own in an overly dramatic wave, a goofy grin on her face. Her inner monologue rolled its eyes at her. “You look like an idiot.” She instantly regretted the action. But everyone seemed to grin and laugh in appreciation.
“They’re laughing at you.”
“Shut up,” she retorted internally, “not tonight.”

Blinking away the tears from the smoke and swallowing down the tingling nerves, she forced herself to walk. Three steps in, and she froze, through the haze, one face, in particular, snapped into focus, and she locked up entirely.

He was talking excitedly, raising a beer, nodding his head, his free hand moved like a conductor, mapping out whatever he was saying. He, unlike the others, seemed oblivious to everything and everyone around him, his audience Mark and a girl, an attractive one she noted with a pang of jealously, nodding along with him.

He wasn’t always here, and she’d only met him a few times. But she’d known two things immediately, one, she liked him, more than just for the night, he made her laugh, and two, she relaxed around him. Something so rare with a guy she’d sort of forgotten it was possible. She let a practiced grin split her face and aimed to strut with what she hoped looked like a confident, sexy walk up to the table.

Men tried to subtly watch her as she moved by, failing spectacularly. She felt elated, giddy, but every ounce of tension that had faded ratcheted up higher and higher as she approached the table. At the last moment, before reaching the table. She stood over and rested her hand on his upper arm to get his attention. He started and turned in surprise his eyes locking onto her’s, a big smile spreading on his face.

“Hey! Pippa, long time no see”
“Hey,” her shoulders relaxing, “great to see you too…”

=============
A parting

“I’m telling you this complete, fucking bullshit,”
“I…”
“Bullshit!”
“Look, I’m sorry, I promise you, this is just a temporar..”
“Bullshit!”

He recoiled as if slapped, the anger in her voice tearing through his words and chest like a hammer. He felt himself shrink inside, shoulders bunched and aching from tension, his stomach churned and spat acid as his traitorous mouth dried up in a panic.

His eyes focused and locked onto her forearm tattoo of a cat. Desperate to concentrate on something but also terrified to take on the full heat of her gaze, even then, he could feel her anger radiating off her in waves. Each thundering heartbeat in his ears boomed with guilt and shame.

“You know that I love you,” he started to say, hating the whining quality that had filtered into his voice.

“Do, you?!” he saw her knuckles whiten, “Because you pay more attention to that fucking thing than me!”
Her finger angrily stabbing at his computer screen.

Petulant resentment flared.
“I’m studying to get a better job! it’ll help us both out of this situation!” he tried to meet her eye.
“It’s not going to be forever, then we can move to a nicer place, a better place.”
Pippa snorted, her lip curling up. She stood over him as he sat coiled in the chair, every nerve and muscle simultaneously ice and fire.
“Do you know how long you’ve been saying that?”
“…”

“I’m done, James, I’m leaving, I’m going to walk out that door, and if I do, you won’t see me again.” her eyes were fire. “Do. You. Understand?”

The silence lasted a breath, then two. A million responses left unsaid.

Then in an instant that he would replay forever, she was turning and striding away, her back straight, fists clenched at her sides, rage and pain flowed out from her body language like a cloud. The edge of her tattoos seemed to stand out like sharp lines against her pale skin, like even they were taught with anger and disgust at him.

His bedroom door slammed so hard behind her that the frame shuddered. For a moment, his vision blurred, the sound of her heavy boots marching off down the corridor.

He heard a strangled noise leave his throat, even as his body slumped in defeat as he listened to the finality of the front door being slammed shut.

He stared blankly at the door for an unknowable amount of time. Part of him screaming to get up and run after her, part of him saying that this wasn’t his fault. His neck creaked and popped when he finally turned to look at the computer screen, the software problem still waiting for him.

He forced himself to rotate the chair. To focus. He sat at his computer, hunched and alone. Here was something he could resolve. No pain, no sacrifice or compromise, desperately trying to ignore the smiling faces of the couple staring accusingly at him from his desktop background, he started to type.