Prerequisites:
The Basics
Overview:
Here are some random "tips & tricks" for working with brushes.
Read up and avoid some of the common pitfalls, such as windows that won't
break, textures on doors that are scaled weird, lighting anomalies, poor performance,
and unexpected "holes".
Contents:
1. Do not scale the initial brushes before making a mover
2. Properly placing the pivot point of mover brushes
3. Avoiding "black door" syndrome
4. Using semi-solid brushes to boost performance
5. Avoiding "holes" in complex brushes
6. Controlling the order of CSG brushes
7. Fixing button slippage on movers
One of the most frustrating things that can happen when creating mover brushes is when you end up with a nonfunctional DeusExMover, such as a door that won't open or a window that won't break. I discovered the hard way that scaling your initial brush (or brushes) with any of the brush scaling tools is usually the cause of the problems.
Though it's a royal pain in the rear, get in the habit of only using the properties windows (such as Cube Properties) to size the red brush before creating your initial subtraction and addition brushes. If you DO scale one of your initial brushes after it's been created and then do your intersect and create your mover, your mover will likely be screwed up beyond repair.
To see if a mover brush has been scaled, right-click on it and pull up its properties. Expand Brush -> MainScale -> Scale and Brush -> PostScale -> Scale. If all of the X, Y, and Z values are not 1.000, you have a problem. Unfortunately, the only way to fix it is to either change the scale values back to 1.000 (almost never what you want, since you probably scaled it for a reason!) or build your initial brush or brushes from scratch, and build another mover.
To get a door to work properly, it's important to know how to set up its pivot point correctly. Moving the pivot point on a mover after it's been created is not a good idea as it can cause weird texture scaling problems.
The technique for setting up the position of the mover's pivot point is easier to show than describe:
 |
 |
| This is an overhead 2D view in UnrealEd of an addition brush ready to be used as a door template for a DeusExMover. The red brush is positioned so that ITS pivot point lines up with where the mover's pivot point will be at the door's "hinge". | After the "intersect" is done and the mover created, you'll see that the mover brush is ready to be used as a swinging door, due to the proper placement of the pivot point. |
Do NOT move the pivot point of a mover brush after it is created. You will run into weird texture scaling issues and swinging doors may act funny. Here's an example of texture weirdness on a mover. The first brush shown here is the original addition brush, the second is a "clean" mover based on that brush, and the third is another mover based on the original brush, but with its pivot point modified after the mover was created. It seems that the texture scaling is reset when you do that:
Ever run into this one?
 |
 |
| This side of the door looks fine, since this room is lit. This is the Keyframe 0 (closed) position. | But the room on the other side is not lit, so when the door is opened, it looks weird. This is the Keyframe 1 (open) position. |
There are three main ways of fixing this. Which solution you choose depends on the situation and what you're trying to achieve.
Solution 1: The reason the back of the door is black is that by default, the surfaces of a mover brush (and any surfaces on nearby geometry) are lit based on the brush's position at Keyframe 0. So an easy solution if your mover is a door to a locker or it's a drawer or something, is to simply put a light inside the locker or drawer.
Solution 2: If you don't actually want to add extra lighting behind the mover, you can instead alter which keyframe is used to light the mover's surfaces as well as surrounding surfaces. Right-click on your mover brush and expand Mover. Set the WorldRaytraceKey to the keyframe to use as the lighting reference. For this example, setting it to 1 will cause the door to be lit in its "open" position, even though the door will initially be closed. It will also cause some light to bleed into the other room, which might look a little weird when the door is shut.
NOTE: There's also a BrushRaytraceKey value you can set, which I think is supposed to only affect the brush. I've had limited success with that though. If you know a way to coerce that setting to work, let me know, as that could be quite handy.
 |
| With the WorldRaytraceKey set to 1, the other side of the door is now lit, but so is some of the other room, regardless of whether the door is open or closed. Could be another good solution for a locker or drawer, but it may not be so great for a room. |
Solution 3: An interesting alternative is to set up your mover to react to lighting dynamically. For this example, I think that will work the best, though it may mean adding an extra light if we want BOTH sides of the door lit when it is open. (it's hard to light two sides of a thin object with a single light) Just pull up the mover brush's properties and set the Mover -> bDynamicLightMover to True. Now the mover will react to normal lights dynamically and will look more realistic, without affecting the lighting in the other room.
 |
 |
| Ah, that's more like it. Now the side of the door facing the dark room will fade to black when the door is closed and light up when it's open. |
Semi-solid brushes are very similar to "additive brushes", but with one crucial difference. They don't create what are called "BSP cuts", which reduces the "node count" of your map. Basically, they are used to make your map more efficient, which boosts your frame rate. They still appear exactly the same to the player, and will still block the player and pawns.
For the same reason semi-solid brushes boost performance, they cannot be used as "subtract" brushes, nor can anything be subtracted from them. So only single brushes are good candidates to make into semi-solid brushes.
While you can make semi-solid brushes with the Add Special button, I prefer to simply right-click on an existing additive brush and select "Make Semisolid" from the Solidity menu. You can always switch a brush back if you run into any issues.
Some examples where you might consider using semi-solid brushes:
Support beams
Roof trusses
Electrical conduits
Any single decorative brush
Legs of a table
Office cubicle walls
Some other general guidelines:
* For some reason, if you have two semi-sold brushes that touch each other,
weird things can happen, so try to avoid that.
* If you have a large, odd-shaped brush that is meant to support the player,
sometimes it's better not to make that a semi-solid brush. I once
had a large disc-shaped semi-solid brush that the player could walk on, and
every once in a while it was possible to "fall through" the brush.
After you start making more complicated brushes, such as terrain and rocks and that sort of thing, you may run into a problem where sections of your brushes appear invisible when your map is played. While these types of "holes" can reportedly be caused by too many polygons (or perhaps too many subtract brushes?) in your map, I've run into other situations that have nothing to do with the number of polygons or brushes.
First of all, if any polygons in a brush are "flipped" the wrong way, you'll be able to see right through them. I purposely flipped a couple of polygons in a model using a 3D modeling package and then imported the model as a brush into UnrealEd to illustrate this. If you import brushes from .DXF files, that's something to look out for. Most modeling software includes tools for viewing and fixing that situation.
 |
| Watch out for "flipped" polygons if you import brushes from .DXF files. |
I once ran into a situation where a brush used for a small hill had several of these holes in it for no apparent reason. The polygons were NOT flipped the wrong way. I had set up the start of the hill to be flush with the "ground" that was created by a subtract brush, and for some reason, that caused problems. After much experimentation, what finally fixed it was lifting the edge up a single unit. It seems that if too much of a multi-polygon brush (such as those created by the floor lofter) is touching another brush, strange things can happen. It also seems to help if the polygons are close to evenly spaced.
 |
| This is a side view of the edge of a "hill" brush zoomed in as far as possible. Lifting the edge of the hill up 1 unit fixed the holes problem and isn't really noticable in the game. |
Some of these holes are quite mysterious, so if anyone has any additional insight, please let me know!
NOTE: If you're still running into holes, check out this BSP FAQ that has more information about these, which are sometimes called "BSP holes." There's also a lot of other good stuff in there about brushes and rebuilding geometry, etc.
I'm not going to go into a lot of detail here, but something to be aware of it is that you can control the "order" that CSG brushes operate. What's CSG? It stands for Constructive Solid Geometry, and is just a fancy way of describing the process of adding, subtracting, unioning, and intersecting 3D shapes to form other shapes. (it comes from the 3D animation/modeling world) In UnrealEd, it just refers to creating level geometry using addition and subtraction brushes.
Anyway, you may run into situations where you want, say, an additive brush to add geometry to your map, but you find that a subtraction brush is causing part of it to be carved out. If you want the subtraction to happen FIRST, you can change the order that the adds and subtracts occur by pulling up one of the brush's properties and selecting either "To first" or "To last" from the Order menu.
As you may know, it's possible to "attach" an actor to a mover brush by setting its Movement -> AttachTag value to the tag name of the mover. Most of the time that works great. For example, you can attach buttons to a mover when creating an elevator.
A problem that doesn't usually crop up when doing a vertical elevator is a little glitch that causes attached actors to slip a little when the mover brush is in motion. If you're doing something like a subway car though, this glitch can happen quite easily.
The problem seems to be that the actor gets hung up a little when the player is standing in the way of the object's path. To fix this, set the buttons' (or other actors') Collision -> bBlockPlayers value to False.