Firstly, it's the missing scene depth for the generated mesh, even the material is set to opaque and as simple as possible. Without the depth there, the overlap meshes will have z-fighting all the time which is unacceptable.

I checked and play around with the vertex factory shader, the FPrimitiveViewRelevance and FMeshBatch configurations and nothing works.

Then I tried the material itself, and do find a workaround for it, setting a small number to the "Pixel Depth Offset" seems force it render to scene depth again. Which gives me a clue, so I follow it up in the render pipeline. So without the "Pixel Depth Offset", the vertex factory's indirect draw will disappear in PrePass DDM_AllOpaque 's depth pass. it's because the engine default shader DepthOnlyVertexShader is used in this case and for our vertex factory shader map, it's missing, and then the DepthRendering::GetDepthPassShaders will early return false.

Once find the root cause, the fix is easy, just make sure add the default shader in the vertex factory permutation using MaterialParameters.bIsSpecialEngineMaterial

bool FVertexFactory::ShouldCompilePermutation(const FVertexFactoryShaderPermutationParameters& Parameters)
{
	return MyVertexFactoryPermutation() || Parameters.MaterialParameters.bIsSpecialEngineMaterial;
}

Then the next issue is while moving the camera around or turn on some vertex animation for the vertex factory generated mesh, part of the mesh will become black and some yellow text behind it saying:

"Your scene contains a skydome mesh with a sky material, but it does not cover that part of the screen..."

not my game, image from internet:

So I believe it's some corrupted buffer for my vertex factory, and I noticed that without the scene depth buffer fix, it's not showing up. After reading the compiled shader I realised that the default DepthOnlyVertexShader still needs call the vertex factory GetVertexFactoryIntermediates, which consume the instance buffer. But the instance buffer is written by a compute shader after the PrePass there. So a classical race condition : )

Then moving that compute shader pass before the PrePass get everyhing works as expected with the vertex factory now.

To simplify, the custom render pass has two parts, the first one is computing shader to generate the instance data in the pre render base pass and then draw these instances in the base pass. For destop platform, it’s very straight forward to add the two custom passes into FDeferredShadingSceneRenderer::Render.

For the mobile platform, the compute pass added into FMobileSceneRenderer::Render and draw pass added into FMobileSceneRenderer::RenderDeferredMultiPass, which works as expected in Unreal editor preview on both Windows and Mac.

But while testing on iOS, it’s not working. Going into the GPU debug in XCode shows that the compute pass is working fine but the draw pass is never called. And then I learnt that for iOS, it’s using FMobileSceneRenderer::RenderDeferredSinglePass.

But adding the draw pass there after the SceneColorRendering pass is still not working on device and GPU debug shows that the draw pass is correctly run and the SceneColor/GBuffer result is as expected. The problem is that it’s too late, because SceneColor/GBuffer is already consumed for back buffer in the SceneColorRendering pass previously (yeah, it’s called for RenderDeferredSinglePass for a reason : ). So the correct way is to make the draw pass a subpass inside the SceneColorRendering pass. Or a better solution is to use unreal vertex factory for drawing our instances, more powerful and flexible, and also it will be handled by FMobileBasePassMeshPassProcessor, which then will be part of the subpass there.

A quick test with a simple vertex factory shows promising result, so I start to fully implement all our features into the vertex factory and then I start getting a crash error while starting the game.

[UE] Ensure condition failed: Texture [File:./Runtime/Renderer/Private/MeshPassProcessor.cpp] [Line: 828] Shader TBasePassVSFNoLightMapPolicy with vertex factory FMyCustomVertexFactory never set texture at BaseIndex 0. This can cause GPU hangs, depending on how the shader uses it.

Which is very wired because I don’t have any texture parameters in my shader. But surprisingly, after checking the compiled shader file for METAL, it turns out somehow my Buffer<uint> ReserveBuffer is translated to texture_buffer ReservedBuffer [[texture(0)]]! Some searching saying this is expected behavior, so for now my fix is to use StructuredBuffer<uint> in my shader.

Now, no crash or errors, but one feature is not working correctly on iOS. And I notice the data structure of this feature is not aligned with 16-bytes.

 struct FMyDataStructure 
{
    FVector3f Pos;
    FVector3f UpVector;
    float Height;
    float Radius;
}

So change it into

struct FMyDataStructure 
{
    FVector4f PosAndHeight;    // .xyz = Pos, .w = Height
    FVector4f UpVectorAndRadius; // .xyz = UpVector, .w = Radius
};

And now everything works on iOS : )

Now I have a perfect test case, and I start debugging it closely. The first thing I noticed is that in the RenderDoc, and in the mesh viewer of my VS pass shows the output SV_Position is shown as NaN (Not a Number) for the almost 1/3 of the instance buffer. The interesting part is that if I debug it step by step, everything just works fine as expected and we got a solid position there. Then I learnt that RenderDoc debug shader is a “simulation” on CPU side, and it cannot guarantee the result match what’s on GPU.

Then I thought it could be some sync issue of the instance buffer or the indirect arguments buffer, wasting a day working on it trying different buffer setup, explicit buffer transition and memory barrieers, no luck.

Ruling out the sync issue, I spent another day to verifying my compute pass to make sure all the data is generated correctly without any surprises.

As I almost gave up, I asked AI if my structure of the instance buffer is cursed : ( but surprisingly, it notice the culprit immediately : )

The real trouble began with a struct that looked like this:

struct FData {
    float3 Position;
    float Width;
    // ... other floats ...
    int Payload;
    float4 ControlPos0; // <-- The Saboteur
};

The structure is 92 bytes in total.

And according to google, it seems that (for NVIDIA specifically), a float4 isn't just four floats—it is a 16-byte vector entity.

And this seems to be the root cause that some of the instance buffer will read "invalid data".

The Fix: Manual Padding is Your Friend

In fact, in latest Unreal Engine (5.7), they add a new variable "RHI_RAW_VIEW_ALIGNMENT" to check if your structured buffer stride is LeastCommonMultiplier of it while creating the buffer.

So for my case, to fix it, we had to add a simple float Padding; after our int Payload;, we manually aligned to the 16-byte boundary. And then the flickering issue goes away.

The ByteAddressBuffer Escape Hatch

And some search result shows there's other work around for this, but I didn't test it.

If you’re tired of fighting with alignment, there is the ByteAddressBuffer. It treats memory as raw bytes, skipping the "smart" auto-padding of Structured Buffers. It's more manual, but it's the ultimate way to ensure what you send is exactly what you get.

struct VertexOutput
{
  float4 Position : SV_POSITION;
}

And this works just perfect fine.

But things get buggy once I add another variable into the struct, which is still very simple

struct VertexOutput
{
  float4 Position : SV_POSITION;
  float4 Test : TEXCOORD0;
}

The render result shows that the position data is still right, but “test” data is always default.

I have no idea what’s wrong so goes RenderDoc to check step by step, and everything seems just working until the PS stage as the input from VS is all default.

But I double check the output of the VS which is definitely fine, and also, the render result position is right.

Drop all the shaders into AI and ask why it happened, and it can hardly understand the real issue, other than giving possible solutions. So I simplified the question as “what possible causes for a PS only get default output from VS?”, and then I can narrow it down to the “VS-To-PS struct mismatch“. While my struct is so simple I cannot tell how it could mismatch, I suddenly realize it could be the unreal shader complier.

Get into the RenderDoc again and check the VS there, and my guess is correct, in the compiled VS shader, the SV_POSITION is reordered to last position in the struct, but no changes for PS shader. That’s why I get a mismatch here.

A quick search in the Unreal Engine source code shows it’s a bug introduced a while ago which only affect indexed indirect draw and fixed later for the shader compiler, and I’m the unlucky guy who is not using the latest version…

Again this is another disappointing experience using AI while coding with shader in Unreal, another case is months ago while I turned on shader debugging the editor cannot start due to shader validation failed:

Get a shader compilation error when I disable shader optimize for current project (Unreal 5.3)
5 Shader compiler errors compiling global shaders for platform PCD3D_SM6:
Shader debug info dumped to: "/Saved/ShaderDebugInfo/PCD3D_SM6/Global/FShadingFurnaceTestPassPS/0"
Engine/Shaders/Private/ShadingFurnaceTest.usf(): Shader FShadingFurnaceTestPassPS, Permutation 0, VF None:
    error: validation errors
Engine/Binaries/Win64/(6395,6): Shader FShadingFurnaceTestPassPS, Permutation 0, VF None:
    zzz.dxil(6395,6): error: Instructions should not read uninitialized value.
/Engine/Shaders/Private/ShadingFurnaceTest.usf(): Shader FShadingFurnaceTestPassPS, Permutation 0, VF None:
    note: at 'br i1 undef, label %1664, label %1699' in block '#89' of function 'MainPS'.
/Engine/Shaders/Private/ShadingFurnaceTest.usf(): Shader FShadingFurnaceTestPassPS, Permutation 0, VF None:
    Validation failed.
/Engine/Shaders/Private/ShadingFurnaceTest.usf(): Shader FShadingFurnaceTestPassPS, Permutation 0, VF None:
    D3DCompileToDxil failed. Error code: Unspecified error (0x80004005).

I think this is an easy one, so I give the shader file and error message to AI and it tried several times and cannot fix it… Then I have to let it explain the shader line by line and ask if the variable is properly initialized, still no luck, I highlight that any custom struct has to be initialized explicitly, and then it finally find the fix:

-                FDeferredLightData LightData;
+                FDeferredLightData LightData = (FDeferredLightData)0;

But this time I’m not that frustrated, I have found a way to “work” with AI now and it does help me to get more efficient : )

error C2039: 'format_string': is not a member of 'std'

Google results will tell you to make sure you're using C++20 flag in the compiler since it's a new feature. I'm pretty sure it's not the issue since std::format itself works fine in the code. But I double checked anyway and try all solutions I could find online (big mistake, trying to be lazy : ). I checked all the dependencies and VS packages needed. I even re-install Visual Studio 2022.

After wasting half a day (and a good sleep) without any progress, I started to be serious about this error.

So back to the error, what's this std::format_string. According to std::basic_format_string, std::format_string, std::wformat_string - cppreference.com, it's a new feature in C++ 20 standard library and should be in the <format> header file. (Yes, I do explicitly include this header file in the source code and nothing changed).

The definition of it is:

template< class... Args >
using format_string = basic_format_string<char, std::type_identity_t<Args>...>;

Then I open the format header in the MSVC and try to search for it. Of course no exact match, so I just tried the patterns like "basic_format_string". Finally, I find this:

template <class... _Args>
using _Fmt_string = _Basic_format_string<char, type_identity_t<_Args>...>;

Surprise, surprise, MSVC has their own naming different from the standard library : )

Then changing the std::format_string to std:_Fmt_string fix the compilation errors. But why I'm getting this while it seems to work fine for others? I realise it's the MSVC version I'm using, it's 14.34.31933 even though the latest one is 14.36.32532.

Then I reinstall MSVC again and this time to ensure I have the latest version of MSVC installed. Then in 14.36.32532, the format header change it to std::format_string and now the original source code is compiled without any issues.

Lesson learnt : )

ERROR: Source file 'C:\workspace\Binaries\Win64\UnrealEditor.target' is not under 'C:\workspace\UnrealEngine'
       while executing task <Copy Files="#InternalBinaries" From="C:\workspace\UnrealEngine" To="C:\workspace\UnrealEngine\LocalBuilds\ArchiveForUGS\Staging" Overwrite="True" ErrorIfNotFound="False" />
       at Engine/Build/Graph/Examples/CustomBuildEditorAndTools.xml(151)
       (see C:\workspace\UnrealEngine\Engine\Programs\AutomationTool\Saved\Logs\Log.txt for full exception trace)

So this is the line causing the error

<Tag Files="#OutputFiles" Except=".../Intermediate/..." With="#ArchiveFiles"/>
<Tag Files="#ArchiveFiles" Except="*.pdb" With="#ArchiveBinaries"/>
<Copy Files="#ArchiveBinaries" From="$(RootDir)" To="$(ArchiveStagingDir)"/>

The #ArchiveBinaries are simply all the filtered files output from the compiling. $(RootDir) is the engine folder (C:\workspace\UnrealEngine) and $(ArchiveStagingDir) is some target folder that doesn't matter. The copy should remain the relative strcutre here, saying C:\workspace\path\to\file should be copied to $(ArchiveStagingDir)\path\to\file

And the error is basically saying that some files is not under the RootDir and can't be copied with relative structure remaining, which makes sense, because you can't have something like $(ArchiveStagingDir)\..\path\to\file.

And the reason why we get files not under the RootDir is simple, because we don't put our game folder as a subfolder of Unreal Engine source. Once the reason is clear, the solution is simple and straight forward:

<Option Name="MyRootDir" DefaultValue="$(RootDir)\..\" Description ="The root dirctory contain both Engine and Game folders"/>
<Copy Files="#ArchiveBinaries" From="$(MyRootDir)" To="$(ArchiveStagingDir)"/>

What we learn today, Unreal want you to put your game under Engine folder if you use the source code, but you can choose to not listen to them :)

UToolMenu* SourceControlMenu = UToolMenus::Get()->ExtendMenu("ID_OF_MENU");
FToolMenuSection& Section = SourceControlMenu->AddSection("MySection");
Section.AddEntry(//my custom menu here//);

So I jump to this work and made the first mistake, I didn't copy paste code and think I'm smart to ignore unreal style guide to just do it in my way.

auto SourceControlMenu = UToolMenus::Get()->ExtendMenu("ID_OF_MENU");
auto Section = SourceControlMenu->AddSection("MySection");
Section.AddEntry(//my custom menu here//);

Compile it, pass in the first try, and run it, breakpoint hitted, it runs my code correctly without any issues, cool, seems work done.

But wait, where is my menu, it's not showing up on the UI. What's going wrong here?

...

After a bad sleep following several hours debugging, searching and asking around online and pulling my hairs, I just realise it's a so simple mistake I'm making here. Looking at the function again

struct FToolMenuSection;
FToolMenuSection& UToolMenu::AddSection(...)

So the return type is a reference, and since it's a struct, then you can implicitly copy the value through a struct reference, so what I'm doing there is making a copy of the section.

I know this is not right to some people, as I quote "I would totally expect the auto to be of the same type of the return type i.e. a reference to FToolMenuSection" (Maybe that's why Unreal suggest to not use auto if possible)

To make it clear:

FToolMenuSection& OriginSection = SourceControlMenu->AddSection("MySection");
auto CopiedSection = OriginSection; //it's not the original section.
// Type of OriginSection is FToolMenuSection&
// Type of CopiedSection is FToolMenuSection

So I'm adding my custom menu into a local variable section which is disposed right after my function...

Besides the mistake I made here, is there a way to protect us from this? yes, there is, you can explicitly delete the constructor of copying from reference

struct FToolMenuSection
{
   FToolMenuSection(FToolMenuSection& ) = delete;
   FToolMenuSection(FToolMenuSection&& ) = delete;
}
FToolMenuSection& OriginSection = SourceControlMenu->AddSection("MySection");
auto CopiedSection = OriginSection; //compiler errors here, don't know how to copy from a reference here

And I tried to do this for Unreal Editor, but can't. Since in the TArray implementation, they need insert fucntion working with reference which needs the copy by reference.

So what I learn today. Always copy paste code where you can :). And maybe follow the style guideline where you can ~

Generally speaking, this could be as simple as:

1. a regular task to check the server.
2. Download the extension package if newer version available.
3. Install that extension package.

You can have all kinds of different impelementations for step 1 or 2, I wouldn't go into details here.

The challenge part is step 3. Previously I just ask my colleagues to manually install the vsix file I shared with them, since it's just several simple clicks on UI. But after 3 releases (bugs/feature request) in a week, this process becomes a bit unacceptable. I have to figure out a way to upgrade it automatically to prevent the fragmentations and also make other's life a bit happier.

So the first question is can we install it through a script/cli? Well, the answer is not that obvious to me (due to my poor google skills), but hey, vscode is opensource, all the answers are already there in the code. So I go to github and quickly go through the issues (always assume somebody else may have the very same issue before), and I find someone report a bug of this command:

code --install-extension path_to_extension_vsix --force

wait, I don't know that and also, it's very clear then it's doable. Everything follows up is simple and boring:

• a lot more help info through code --help.
• searching through the code by "install-extension"
• ha, cli command: "workbench.extensions.installExtension" (in fact, it's right there in the offical documentation :))
• basically just call this vs command in the extension code with the vsix file URI, it's done.

Can't believe I go through such a long way (again, poor google skills), but a lot of other stuffs learnt.