C ++를 컴파일하기 위해 Windows를 Linux만큼 빠르게 실행하려면 어떻게해야합니까?
나는 이것이 프로그래밍 문제가 아니라 관련성이 있다는 것을 알고있다.
저는 상당히 큰 크로스 플랫폼 프로젝트를 진행하고 있습니다. Windows에서는 VC ++ 2008을 사용합니다. Linux에서는 gcc를 사용합니다. 프로젝트에는 약 40k 개의 파일이 있습니다. Windows는 동일한 프로젝트를 컴파일하고 링크 할 때 Linux보다 10 배에서 40 배 정도 느립니다. 어떻게 고칠 수 있습니까?
Linux에서 단일 변경 증분 빌드 20 초, Windows에서> 3 분 왜? 리눅스에 '골드'링커를 설치하고 그 시간을 7 초로 줄일 수도 있습니다.
마찬가지로 git은 Windows보다 Linux에서 10x ~ 40x 빠릅니다.
자식의 경우 git이 Windows를 최적의 방식으로 사용하지 않고 VC ++을 사용할 수 있습니까? 마이크로 소프트는 자신의 개발자가 가능한 한 생산성을 높이기를 원한다고 생각할 것이고, 더 빠른 컴파일은 그와는 거리가 멀다. 어쩌면 그들은 개발자를 C #으로 격려하려고 노력하고 있습니까?
간단한 테스트로서 많은 하위 폴더가있는 폴더를 찾아서
dir /s > c:\list.txt
Windows에서. 두 번 실행하고 두 번째 실행 시간을 지정하여 캐시에서 실행하십시오. 파일을 Linux에 복사하고 동등한 2 회 실행과 두 번째 실행 시간을 수행하십시오.
ls -R > /tmp/list.txt
정확히 동일한 사양을 가진 2 대의 워크 스테이션이 있습니다. 12gig 램, 3.0ghz에서 8 코어의 HP Z600 ~ 400k 파일이있는 폴더에서 Windows는 40 초, Linux는 1 초 미만입니다.
Windows 속도를 높이기 위해 설정할 수있는 레지스트리 설정이 있습니까? 무엇을 제공합니까?
컴파일 시간과 관련된 약간의 관련 링크는 반드시 i / o 일 필요는 없습니다.
분명히 Windows 10 (Windows 7 아님)에는 프로세스를 닫으면 전역 잠금이 유지되는 문제가 있습니다. 여러 코어로 컴파일 할 때 여러 프로세스를 컴파일 할 때이 문제가 발생합니다.
이
/analyse옵션은 웹 브라우저를로드하기 때문에 성능에 악영향을 줄 수 있습니다 . (여기에는 관련이 없지만 알아두면 좋은 점)
하드 코어 Windows 시스템 해커가 등장하지 않는 한, 당신은 당파적인 의견 (내가하지 않을 것)과 추측 (내가 시도 할 것) 이상을 얻지 못할 것입니다.
파일 시스템-동일한 파일 시스템에서 동일한 작업 (을 포함하여
dir)을 시도해야합니다 . 다양한 매개 변수에 대해 몇 가지 파일 시스템을 벤치마킹하는 이 문제 를 발견 했습니다 .캐싱. 한때 RAM 디스크에서 Linux에서 컴파일을 시도했지만 커널이 캐싱을 처리하는 방식으로 인해 디스크에서 실행하는 것보다 느리다는 것을 알았습니다. 이것은 Linux의 확실한 판매 지점이며 성능이 다른 이유 일 수 있습니다.
Windows의 종속성 사양이 잘못되었습니다. Windows의 크롬 종속 사양이 Linux만큼 정확하지 않을 수 있습니다. 작은 변경을하면 불필요한 컴파일이 발생할 수 있습니다. Windows에서 동일한 컴파일러 툴체인을 사용하여이를 확인할 수 있습니다.
몇 가지 아이디어 :
- 8.3 이름을 비활성화하십시오. 많은 파일과 비교적 적은 수의 폴더가있는 드라이브에서 이는 큰 요인이 될 수 있습니다.
fsutil behavior set disable8dot3 1 - 더 많은 폴더를 사용하십시오. 내 경험상 NTFS는 폴더 당 약 1000 개가 넘는 파일로 속도가 느려집니다.
- MSBuild로 병렬 빌드를 사용하십시오. "/ m"스위치 만 추가하면 CPU 코어 당 하나의 MSBuild 복사본이 자동으로 시작됩니다.
- 파일을 SSD에 저장하면 임의 I / O에 큰 도움이됩니다.
- 평균 파일 크기가 4KB보다 훨씬 큰 경우 평균 파일 크기와 대략적으로 더 큰 클러스터 크기로 파일 시스템을 재구성하는 것이 좋습니다.
- 파일이 조각 모음되었는지 확인하십시오. 조각난 파일은 많은 디스크 검색을 유발하므로 처리량을 40 배 이상 높일 수 있습니다. sysinternals의 "contig"유틸리티 또는 내장 Windows 조각 모음을 사용하십시오.
- 평균 파일 크기가 작고 현재 파티션이 상대적으로 가득 찬 경우 조각난 MFT로 실행 중일 수 있으며 이는 성능에 좋지 않습니다. 또한 1K보다 작은 파일은 MFT에 직접 저장됩니다. 위에서 언급 한 "contig"유틸리티가 도움이되거나 MFT 크기를 늘려야 할 수도 있습니다. 다음 명령은 볼륨을 25 %로 두 배로 늘립니다
fsutil behavior set mftzone 2. 마지막 숫자를 3 또는 4로 변경하여 크기를 추가로 12.5 % 씩 증가시킵니다. 명령을 실행 한 후 재부팅 한 후 파일 시스템을 작성하십시오. - 마지막 액세스 시간을 비활성화하십시오.
fsutil behavior set disablelastaccess 1 - 인덱싱 서비스 비활성화
- 안티 바이러스 및 안티 스파이웨어 소프트웨어를 비활성화하거나 최소한 관련 폴더를 무시하도록 설정하십시오.
- 파일을 OS 및 페이징 파일과 다른 실제 드라이브에 넣습니다. 별도의 물리적 드라이브를 사용하면 Windows가 두 드라이브 모두에 병렬 I / O를 사용할 수 있습니다.
- 컴파일러 플래그를 살펴보십시오. Windows C ++ 컴파일러에는 수많은 옵션이 있습니다. 당신이 정말로 필요한 것을 사용하고 있는지 확인하십시오.
- OS가 페이지 풀 버퍼에 사용하는 메모리 양을 늘리십시오 (먼저 RAM이 충분한 지 확인하십시오).
fsutil behavior set memoryusage 2 - 때때로 디스크 오류가 발생하지 않도록 Windows 오류 로그를 확인하십시오.
- 실제 디스크 관련 성능 카운터를보고 디스크 사용량이 많은지 확인하십시오. 큐 길이가 길거나 전송 당 시간이 길면 잘못된 신호입니다.
- 디스크 파티션의 처음 30 %는 원시 전송 시간 측면에서 나머지 디스크보다 훨씬 빠릅니다. 좁은 파티션은 탐색 시간을 최소화하는 데 도움이됩니다.
- RAID를 사용하고 있습니까? 그렇다면 RAID 유형 선택을 최적화해야 할 수도 있습니다 (RAID-5는 컴파일과 같이 쓰기가 많은 작업에는 좋지 않습니다)
- 필요하지 않은 서비스 비활성화
- 폴더 조각 모음 : 모든 파일을 다른 드라이브 (파일 만)로 복사하고, 원본 파일을 삭제하고, 모든 폴더를 다른 드라이브 (빈 폴더 만)로 복사 한 다음 원래 폴더를 삭제하고, 원본 드라이브의 조각 모음을 수행하고, 폴더 구조를 먼저 복사하십시오. 파일을 복사하십시오. Windows에서 한 번에 하나의 파일로 큰 폴더를 만들면 폴더가 조각화되고 느려집니다. ( "contig"도 여기에 도움이됩니다)
- I / O에 바인딩되어 있고 여분의 CPU주기가있는 경우 디스크 압축을 켜십시오. CPU 비용이 약간 들지만 압축률이 높은 파일 (예 : 소스 코드)의 속도를 크게 높일 수 있습니다.
NTFS saves file access time everytime. You can try disabling it: "fsutil behavior set disablelastaccess 1" (restart)
The issue with visual c++ is, as far I can tell, that it is not a priority for the compiler team to optimize this scenario. Their solution is that you use their precompiled header feature. This is what windows specific projects have done. It is not portable, but it works.
Furthermore, on windows you typically have virus scanners, as well as system restore and search tools that can ruin your build times completely if they monitor your buid folder for you. windows 7 resouce monitor can help you spot it. I have a reply here with some further tips for optimizing vc++ build times if you're really interested.
I personally found running a windows virtual machine on linux managed to remove a great deal of the IO slowness in windows, likely because the linux vm was doing lots of caching that Windows itself was not.
Doing that I was able to speed up compile times of a large (250Kloc) C++ project I was working on from something like 15 minutes to about 6 minutes.
The difficulty in doing that is due to the fact that C++ tends to spread itself and the compilation process over many small, individual, files. That's something Linux is good at and Windows is not. If you want to make a really fast C++ compiler for Windows, try to keep everything in RAM and touch the filesystem as little as possible.
That's also how you'll make a faster Linux C++ compile chain, but it is less important in Linux because the file system is already doing a lot of that tuning for you.
The reason for this is due to Unix culture: Historically file system performance has been a much higher priority in the Unix world than in Windows. Not to say that it hasn't been a priority in Windows, just that in Unix it has been a higher priority.
Access to source code.
You can't change what you can't control. Lack of access to Windows NTFS source code means that most efforts to improve performance have been though hardware improvements. That is, if performance is slow, you work around the problem by improving the hardware: the bus, the storage medium, and so on. You can only do so much if you have to work around the problem, not fix it.
Access to Unix source code (even before open source) was more widespread. Therefore, if you wanted to improve performance you would address it in software first (cheaper and easier) and hardware second.
As a result, there are many people in the world that got their PhDs by studying the Unix file system and finding novel ways to improve performance.
Unix tends towards many small files; Windows tends towards a few (or a single) big file.
Unix applications tend to deal with many small files. Think of a software development environment: many small source files, each with their own purpose. The final stage (linking) does create one big file but that is an small percentage.
As a result, Unix has highly optimized system calls for opening and closing files, scanning directories, and so on. The history of Unix research papers spans decades of file system optimizations that put a lot of thought into improving directory access (lookups and full-directory scans), initial file opening, and so on.
Windows applications tend to open one big file, hold it open for a long time, close it when done. Think of MS-Word. msword.exe (or whatever) opens the file once and appends for hours, updates internal blocks, and so on. The value of optimizing the opening of the file would be wasted time.
The history of Windows benchmarking and optimization has been on how fast one can read or write long files. That's what gets optimized.
Sadly software development has trended towards the first situation. Heck, the best word processing system for Unix (TeX/LaTeX) encourages you to put each chapter in a different file and #include them all together.
Unix is focused on high performance; Windows is focused on user experience
Unix started in the server room: no user interface. The only thing users see is speed. Therefore, speed is a priority.
Windows started on the desktop: Users only care about what they see, and they see the UI. Therefore, more energy is spent on improving the UI than performance.
The Windows ecosystem depends on planned obsolescence. Why optimize software when new hardware is just a year or two away?
I don't believe in conspiracy theories but if I did, I would point out that in the Windows culture there are fewer incentives to improve performance. Windows business models depends on people buying new machines like clockwork. (That's why the stock price of thousands of companies is affected if MS ships an operating system late or if Intel misses a chip release date.). This means that there is an incentive to solve performance problems by telling people to buy new hardware; not by improving the real problem: slow operating systems. Unix comes from academia where the budget is tight and you can get your PhD by inventing a new way to make file systems faster; rarely does someone in academia get points for solving a problem by issuing a purchase order. In Windows there is no conspiracy to keep software slow but the entire ecosystem depends on planned obsolescence.
Also, as Unix is open source (even when it wasn't, everyone had access to the source) any bored PhD student can read the code and become famous by making it better. That doesn't happen in Windows (MS does have a program that gives academics access to Windows source code, it is rarely taken advantage of). Look at this selection of Unix-related performance papers: http://www.eecs.harvard.edu/margo/papers/ or look up the history of papers by Osterhaus, Henry Spencer, or others. Heck, one of the biggest (and most enjoyable to watch) debates in Unix history was the back and forth between Osterhaus and Selzer http://www.eecs.harvard.edu/margo/papers/usenix95-lfs/supplement/rebuttal.html You don't see that kind of thing happening in the Windows world. You might see vendors one-uping each other, but that seems to be much more rare lately since the innovation seems to all be at the standards body level.
That's how I see it.
Update: If you look at the new compiler chains that are coming out of Microsoft, you'll be very optimistic because much of what they are doing makes it easier to keep the entire toolchain in RAM and repeating less work. Very impressive stuff.
Incremental linking
If the VC 2008 solution is set up as multiple projects with .lib outputs, you need to set "Use Library Dependency Inputs"; this makes the linker link directly against the .obj files rather than the .lib. (And actually makes it incrementally link.)
Directory traversal performance
It's a bit unfair to compare directory crawling on the original machine with crawling a newly created directory with the same files on another machine. If you want an equivalent test, you should probably make another copy of the directory on the source machine. (It may still be slow, but that could be due to any number of things: disk fragmentation, short file names, background services, etc.) Although I think the perf issues for dir /s have more to do with writing the output than measuring actual file traversal performance. Even dir /s /b > nul is slow on my machine with a huge directory.
I'm pretty sure it's related to the filesystem. I work on a cross-platform project for Linux and Windows where all the code is common except for where platform-dependent code is absolutely necessary. We use Mercurial, not git, so the "Linuxness" of git doesn't apply. Pulling in changes from the central repository takes forever on Windows compared to Linux, but I do have to say that our Windows 7 machines do a lot better than the Windows XP ones. Compiling the code after that is even worse on VS 2008. It's not just hg; CMake runs a lot slower on Windows as well, and both of these tools use the file system more than anything else.
The problem is so bad that most of our developers that work in a Windows environment don't even bother doing incremental builds anymore - they find that doing a unity build instead is faster.
Incidentally, if you want to dramatically increase compilation speed in Windows, I'd suggest the aforementioned unity build. It's a pain to implement correctly in the build system (I did it for our team in CMake), but once done automagically speeds things up for our continuous integration servers. Depending on how many binaries your build system is spitting out, you can get 1 to 2 orders of magnitude improvement. Your mileage may vary. In our case I think it sped up the Linux builds threefold and the Windows one by about a factor of 10, but we have a lot of shared libraries and executables (which decreases the advantages of a unity build).
How do you build your large cross platform project? If you are using common makefiles for Linux and Windows you could easily degrade windows performance by a factor of 10 if the makefiles are not designed to be fast on Windows.
I just fixed some makefiles of a cross platform project using common (GNU) makefiles for Linux and Windows. Make is starting a sh.exe process for each line of a recipe causing the performance difference between Windows and Linux!
According to the GNU make documentation
.ONESHELL:
should solve the issue, but this feature is (currently) not supported for Windows make. So rewriting the recipes to be on single logical lines (e.g. by adding ;\ or \ at the end of the current editor lines) worked very well!
IMHO this is all about disk I/O performance. The order of magnitude suggests a lot of the operations go to disk under Windows whereas they're handled in memory under Linux, i.e. Linux is caching better. Your best option under windows will be to move your files onto a fast disk, server or filesystem. Consider buying an Solid State Drive or moving your files to a ramdisk or fast NFS server.
I ran the directory traversal tests and the results are very close to the compilation times reported, suggesting this has nothing to do with CPU processing times or compiler/linker algorithms at all.
Measured times as suggested above traversing the chromium directory tree:
- Windows Home Premium 7 (8GB Ram) on NTFS: 32 seconds
- Ubuntu 11.04 Linux (2GB Ram) on NTFS: 10 seconds
- Ubuntu 11.04 Linux (2GB Ram) on ext4: 0.6 seconds
For the tests I pulled the chromium sources (both under win/linux)
git clone http://github.com/chromium/chromium.git
cd chromium
git checkout remotes/origin/trunk
To measure the time I ran
ls -lR > ../list.txt ; time ls -lR > ../list.txt # bash
dir -Recurse > ../list.txt ; (measure-command { dir -Recurse > ../list.txt }).TotalSeconds #Powershell
I did turn off access timestamps, my virus scanner and increased the cache manager settings under windows (>2Gb RAM) - all without any noticeable improvements. Fact of the matter is, out of the box Linux performed 50x better than Windows with a quarter of the RAM.
For anybody who wants to contend that the numbers wrong - for whatever reason - please give it a try and post your findings.
Try using jom instead of nmake
Get it here: http://qt.gitorious.org/qt-labs/jom
The fact is that nmake is using only one of your cores, jom is a clone of nmake that make uses of multicore processors.
GNU make do that out-of-the-box thanks to the -j option, that might be a reason of its speed vs the Microsoft nmake.
jom works by executing in parallel different make commands on different processors/cores. Try yourself an feel the difference!
I want to add just one observation using Gnu make and other tools from MinGW tools on Windows: They seem to resolve hostnames even when the tools can not even communicate via IP. I would guess this is caused by some initialisation routine of the MinGW runtime. Running a local DNS proxy helped me to improve the compilation speed with these tools.
Before I got a big headache because the build speed dropped by a factor of 10 or so when I opened a VPN connection in parallel. In this case all these DNS lookups went through the VPN.
This observation might also apply to other build tools, not only MinGW based and it could have changed on the latest MinGW version meanwhile.
I recently could archive an other way to speed up compilation by about 10% on Windows using Gnu make by replacing the mingw bash.exe with the version from win-bash
(The win-bash is not very comfortable regarding interactive editing.)
'Programming' 카테고리의 다른 글
| 왜 서로 다른 두 개념이“힙”이라고 불리는가? (0) | 2020.06.22 |
|---|---|
| JBoss vs Tomcat 다시 (0) | 2020.06.22 |
| 태그와 릴리스의 차이점은 무엇입니까? (0) | 2020.06.22 |
| 선언이 아닌 경우 배열 초기화 구문 (0) | 2020.06.22 |
| 브라우저 간 테스트 : 한 대의 컴퓨터에있는 모든 주요 브라우저 (0) | 2020.06.22 |