scripting languages weren't really designed for large-scale development efforts involving millions of lines of code
Neither was C.
There are languages developed afterwards with large-scale development in mind, but you have to ask, what did they add? True type safety? Nope. True encapsulation? Nope. Automatic resource management? Nope. C++ is C with several times more ways to write horrifyingly unmaintainable code. All big C++ shops have a very long lists of constructs that must never be used and very rigorous code reviews to ensure that you don't use any of those features by accident.
At this point, we see that C++ has one advantage over C, and that's namespaces. Helpful, but not helpful enough for "large-scale software development". The major innovation that C++ brought was waking people up to the reality that a maintainable codebase must be curated with extensive automatic testing and manual code reviews. Anything else leads to epic failure.
But wait, that's easy to fix! Let's invent a new programming language! This time we'll call it Java. It will be like C++ but with all the ways to write bad code removed. No operator overloading! No multiple inheritance.
And it's true that Java helped in a number of ways. But it didn't solve the real problems. There is still no type safety; null is an instance of every class in Java, but you can't call any methods on it, for example. So instead of a segfault, you get a NullPointerException, but all that means is that the source of the error is easier to determine. But you still have to write a lot of tests to make sure that your code handles nulls properly. (This is compounded by laziness in design like writing "loggedInUser = null" instead of writing a subclass of User that indicates it's not logged in.)
Multiple inheritance in C++ was a mess, but Java's solution isn't much better. What's the conceptual difference between abstract classes and interfaces? Abstract classes are non-composable class-parts that contain API, implementation, and state. Interfaces are composable class-parts that contain API. Why are these two separate concepts? Why not have a generic "traits" feature? (The answer is because Java is mostly a copy of C++, and C++ focuses on irrelevant OO features like 4 different levels of member visibility rather than semantic annotations like "if you compose this method into another class, it should run after the method from the class it's being composed into". Of course, when you have these annotations, multiple-inheritance or multiple-trait-application works perfectly -- see CLOS or Moose. But all C++ had was public/protected/private/friend, and so that's all Java has.)
This is getting ramble-y so I'll get to the point. No modern programming language helps you write huge codebases. If you want to be able to maintain millions of lines of code, you are going to need very rigorous standards and millions of lines of automatic tests. It's the only way we know of. All more modern languages did was push us from "we're leaking memory because we forgot to call free()" to "we're leaking memory because it's pretty convenient to keep all these huge objects around".
They certainly haven't helped us write more maintainable large projects. If anything, a dynamic scripting language embedded in your game means that you'll have fewer lines of code and clearer separation between components. But it's not such a win that you won't have to review code or write tests anymore.
The D programming language has a number of features that are strongly oriented towards being able to deal with very large code bases:
1. checkable function purity
2. checkable transitive immutability
3. code can be divided into checkably memory safe code and unsafe code
4. anti-hijacking enforcement
5. modules with closed name spaces
6. memory is thread local by default (shared memory is typed differently)
7. look for non-null pointers in the near future
"No modern programming language helps you write huge codebases."
Have you used Go? This is one of our design goals. Go is a really simple language. Its features are easy to understand and predictable in use. Go code is also very readable in that you don't need a lot of context to understand a piece of code. There aren't any colossal Go codebases yet, but so far things are looking promising.
I haven't used Go for anything huge, but it does feel like good progress for the future. (In a comment a few pages down in this thread, I say as much.)
In the end though, bad code is mostly due to bad programming and bad process. Go is obscure enough that the bad programmers haven't heard about it yet. Start offering high-paying Go jobs to anyone with a pulse and you'll start seeing why people hate C++ and Java so much. It's not that the language sucks, it's that the programmers using it do.
It always surprised me to hear that Google "got by" on C++ and Java but I lightened up a bit when I was reading some of Android. Normally you open up Java and are immediately stunned by the smell it's emitting, but when I started reading the Android code, this didn't happen. Classes did one thing and delegated to other classes when they needed something done. The methods were small and made sense. Line of code inside methods were in "paragraphs". There were no comments like "// hack around bug in SomeOtherClassIWroteButAmTooLazyToFix". It was clear that it was the work of someone who knew what she was doing.
I guess I knew it was possible, but was never convinced of it by any concrete code. The standard library, for example, is horrifyingly bad.
Ultimately languages can lead you in the right direction or the wrong direction, but which path you take depends on the programmer. Google requires code reviews for nearly every commit, and they get to hire the top 0.01% of programmers. Imagine what the rest of the world is like, without code reviews, testing, or good programmers.
"Google requires code reviews for nearly every commit, and they get to hire the top 0.01% of programmers. Imagine what the rest of the world is like, without code reviews, testing, or good programmers."
Yet even with our quality of code we are still struggling with our massive code bases. C++ build times alone are reason to find an alternative. We hope that Go will work at scale while having many of the productivity advantages of scripting languages.
> I guess I knew it was possible, but was never convinced of it by any concrete code. The standard library, for example, is horrifyingly bad.
Between your comment about Android and this one, it looks like your problem is much more with the Java API's than Java itself.
Besides, I disagree: the Java collections are fairly solid with the right mix of abstraction and efficiency. Compare with those of Scala, for example, which require a lot more work before they become decent.
> There aren't any colossal Go codebases yet, but so far things are looking promising.
Not really. Go keeps all the old errors everybody should know are errors (nullable pointers, shared mutable state, raw types, ...) and then proceeds to add new ones, and packages all of that in a less regular syntax just in case there was any chance to get a good language out of the previous clusterfuck.
Nullable pointers and shared state reflect the way the machine actually works. If you regard these design decisions as mistakes, then Go clearly isn't the language for you.
(also, which language is Go less regular than? Lisp?)
Go strikes a middle ground between low-level and high-level in some awkward ways. It wouldn't be hard to use e.g. nullable types or option types to outlaw null pointer exceptions without restricting the set of possible programs, and with stronger static guarantees of correctness. On the other hand, Go also has mandatory garbage collection, which emphatically does not reflect the underlying machine and also restricts its usefulness in certain situations.
w/r/t regularity: most of the functional programming languages (e.g. ML, Haskell sans GHC extensions, various Lisps) are incredibly regular, especially in the semantic sense of providing a few semantically simple features and milking them for all they're worth. Go has quite a few special cases (e.g. the make versus new distinction, the iota keyword) and some odd omissions (e.g. simulating union types involves what I perceive as interface trickery; const only allows numbers or strings as values.) Coming from C++, Java, &c, Go seems incredibly regular—the lack of OOP goes a long way towards keeping it simple—but it's not a simple language except in the context of "modern, Algol-derived applications languages." Which it is an improvement on, but it's not regular in the strict sense.
> Go keeps all the old errors everybody should know are errors (nullable pointers, shared mutable state, raw types, ...) and then proceeds to add new ones,
And because of the absence of exceptions, Go forces you to deal with errors at the call site (see the number of times you see "ok, err = Foo(); if (err)..." which is not scalable to large scale software.
I left this specific point out, because I'm on the fence about it. I do think the C-style way of Go is a genuine mistake, but I also think when type systems are used to force the caller to know about what's happening but the language provides tool which let this be done in a non-absolutely-painful manner (à la haskell, with the `Either` type being used to report success/error, and pattern matching or monadic lifting letting users either act cleanly or propagate errors without being overly verbose and drowning their own code in explicit error propagation) it works rather well, and limits the amount of runtime surprises.
On the other hand, return-value-error-reporting does not give a way for deep callers (caller of the original API when the error happens 6 frames down the stack) to try and recover (instead of just bail out, or more generally customize the error recovery policy) the way condition systems do in Smalltalk, Common Lisp or Dylan.
From the Article: scripting languages weren't really designed for large-scale development efforts involving millions of lines of code
In this instance Carmark is talking about a specific instance - scripting for high-performance games, that still need to work with the tradeoff of high levels of reliability.
He even mentions tempted by functional languages such as Haskell and Caml. Then he lists a couple of reasons it's not appropriate - performance is one, the learning curve is the other.
He then goes on to say that he'd think about it differently, but that performance is such a dominating factor and issue for them. That's driving the majority of the logic.
So I'd not draw too long a bow to suggest that he's canning scripting languages in general. I think the author is stretching with that statement.
you'll have fewer lines of code and clearer separation between components
I think that's the most important statement. If you have millions of lines of code, the answer to to _write less_... and that's where dynamic & functional languages are so important.
I've worked on large codebases. It's rare that one language can cover the whole domain without some big potholes appearing (when I say cover I mean vertically in terms of low-high level code and horizontally in terms of function). That's where your DSL and dynamic languages some into play.
Out of curiosity, what do you think a language geared towards "big software" projects would/should look like and does it exist, yet? Or would you argue, at some point, we hit a design vs. code problem where choice of language doesn't matter so much as does choice of framework and we push more of the boiler plate code onto the framework?
I'm not sure a language can solve all the problems.
One I think would help would be the inability to interface with concrete APIs. If you want to have a variable or instance attribute or argument to a function or method, its type must be declared in terms of an interface, not a particular implementation. This way, you never get yourself into the situation where you say, "I don't want to inherit from Foo, I want to implement its interface myself" but can't.
Beyond that, there is only so much a language can do to help you. It's nice to have guaranteed privacy, but if you use it like the Java standard library does, it becomes the worst language feature ever. To prevent misuse of a powerful feature, you need to be smart and you need to have other smart people reading your code.
The "way forward" is to realize that software development is not easy.
Neither was C.
There are languages developed afterwards with large-scale development in mind, but you have to ask, what did they add? True type safety? Nope. True encapsulation? Nope. Automatic resource management? Nope. C++ is C with several times more ways to write horrifyingly unmaintainable code. All big C++ shops have a very long lists of constructs that must never be used and very rigorous code reviews to ensure that you don't use any of those features by accident.
At this point, we see that C++ has one advantage over C, and that's namespaces. Helpful, but not helpful enough for "large-scale software development". The major innovation that C++ brought was waking people up to the reality that a maintainable codebase must be curated with extensive automatic testing and manual code reviews. Anything else leads to epic failure.
But wait, that's easy to fix! Let's invent a new programming language! This time we'll call it Java. It will be like C++ but with all the ways to write bad code removed. No operator overloading! No multiple inheritance.
And it's true that Java helped in a number of ways. But it didn't solve the real problems. There is still no type safety; null is an instance of every class in Java, but you can't call any methods on it, for example. So instead of a segfault, you get a NullPointerException, but all that means is that the source of the error is easier to determine. But you still have to write a lot of tests to make sure that your code handles nulls properly. (This is compounded by laziness in design like writing "loggedInUser = null" instead of writing a subclass of User that indicates it's not logged in.)
Multiple inheritance in C++ was a mess, but Java's solution isn't much better. What's the conceptual difference between abstract classes and interfaces? Abstract classes are non-composable class-parts that contain API, implementation, and state. Interfaces are composable class-parts that contain API. Why are these two separate concepts? Why not have a generic "traits" feature? (The answer is because Java is mostly a copy of C++, and C++ focuses on irrelevant OO features like 4 different levels of member visibility rather than semantic annotations like "if you compose this method into another class, it should run after the method from the class it's being composed into". Of course, when you have these annotations, multiple-inheritance or multiple-trait-application works perfectly -- see CLOS or Moose. But all C++ had was public/protected/private/friend, and so that's all Java has.)
This is getting ramble-y so I'll get to the point. No modern programming language helps you write huge codebases. If you want to be able to maintain millions of lines of code, you are going to need very rigorous standards and millions of lines of automatic tests. It's the only way we know of. All more modern languages did was push us from "we're leaking memory because we forgot to call free()" to "we're leaking memory because it's pretty convenient to keep all these huge objects around".
They certainly haven't helped us write more maintainable large projects. If anything, a dynamic scripting language embedded in your game means that you'll have fewer lines of code and clearer separation between components. But it's not such a win that you won't have to review code or write tests anymore.