How can I use Python for large-scale development?

Question

I would be interested to learn about the large-scale development in Python, and especially about how you maintain a large code base?

• When you make incompatibility changes with a method signature, how do you find all the places where this method is called. In C ++ / Java, the compiler will find it for you, how do you do it in Python?

• When you make changes inside the code, how do you know what operations the instance provides, since you do not have a static type to search for?

• How do you handle / prevent input errors (typos)?

• Is UnitTest used as a replacement for checking static type?

As you can guess, I almost only worked with statically typed languages ​​(C ++ / Java), but I would like to try my hand at Python for larger programs. But I had a very bad experience, a long time ago, with the clipping language (dBase), which was also dynamically printed.

Answer 1

Since no one pointed to pychecker, pylint and similar tools, I will: pychecker and pylint are tools that will help you find the wrong assumptions (about function signatures, object attributes, etc.). They will not find everything that the compiler can do to find in a statically typed language - but they can find problems that such compilers for such languages ​​can not find either.

Python (and any dynamically typed language) is fundamentally different in terms of the errors you are likely to cause and how to detect and fix them. It has certain flaws as well as problems, but many (including me) argue that in the case of Python, the simplicity of writing code (and the simplicity of its structural sound) and modifying the code without violating API compatibility (adding new optional arguments, providing different objects that have the same set of methods and attributes) make it suitable only for large code bases.

Answer 2

Do not use a screwdriver as a hammer

Python is not a statically typed language, so do not try to use it that way.

When you use a specific tool, you use it to build it. For Python, this means:

• Duck seal : no type check. Only behavior is important. Therefore, your code must be designed to use this feature. Good design means generic signatures, no dependencies between components, high levels of abstraction. Therefore, if you change something, you do not have to change the rest of the code. Python will also not complain about what it was created for. Types are not a problem.

• Huge standard library . You do not need to change all your calls in the program if you use standard functions that you did not code yourself. And Python comes with batteries included. I keep opening them every day. I did not know how many modules I could use when I started, and tried to rewrite existing things, like everyone else. This is normal, you cannot fix everything from the very beginning.

You do not write Java, C ++, Python, PHP, Erlang, regardless of the same. They are good reasons why there is room for each of many different languages, they do not do the same.

Unit tests are not a substitute

Unit tests must be performed in any language. The most famous unit test library ( JUnit ) from the Java world!

This has nothing to do with types. You check the behavior again. You avoid regression issues. You guarantee that your customer is on the tracks.

Python for large-scale projects

Languages, libraries, and frameworks do not scale. Architecture do.

If you create a solid architecture, if you can quickly develop it, then it will scale. The help of unit tests, automatic code verification. But they are just security systems. And small ones.

Python is especially suitable for large projects because it applies some good practices and has many common design patterns. But then again, do not use it to ensure that it is not intended. For example: Python is not a technology for intensive processor tasks.

In a huge project, you are likely to use several different technologies. Like SGBD and template language, otherwise. Python is no exception.

You probably want to use C / C ++ for the part of your code that you need in order to be fast. Or Java to fit in with Tomcat . I don’t know, anyway. Python can play well with them.

In conclusion

My answer may seem a little rude, but don't get me wrong: this is a very good question.

Many people come to Python with old habits. I screwed myself up trying to code Java as Python. You can, but never get the best of it.

If you played / want to play with Python, this is great! This is a great tool. But just a tool, really.

Answer 3

I had experience with the modification "Frets On Fire", the python open source clone "Guitar Hero".

as i see python is not suitable for a really large-scale project.

I found that most of the development time debugs the problems associated with the assignment of incompatible types, that static typed laguations will be easily detected at compile time. also, since types are determined at runtime, trying to understand existing code becomes more difficult because you have no idea what type of parameter you are currently looking at.

in addition to this, calling functions that use their own name string with the __getattr__ built-in function __getattr__ usually more common in Python than in other programming languages, which makes the call graph for a certain function somewhat difficult (although you can call functions with their name in some statically typed languages).

I think Python really shines in small software, rapid prototyping and gluing existing programs together, but I would not use it for large-scale software projects, since maintainability becomes a real problem in those types of programs, and in my opinion python is relatively weak there .

Answer 4

my 0.10 EUR:

I have several python applications in the "production" state. Our company uses java, C ++ and python. we are developing using eclipse ide (pydev for python)

unittests is a key solution to the problem. (also for C ++ and java)

a less secure world of "dynamic input" will make you less careless about the quality of the code

ON THE WAY :

large-scale development does not mean that you use one language!

large-scale development often uses several languages ​​specific to the problem .

so I agree with the hammer problem :-)

---

PS: static typing and python

Answer 5

Here are some elements that helped me maintain a fairly large system in python.

• Structure your code in layers. that is, a separate logic logic, presentation logic and your persistence levels. Spend a little time identifying these layers and make sure everything in the project is purchased. For large systems that create a structure that forces you to evolve in a certain way, can also be key.

• Tests are key, without unit tests you will most likely end up with an unmanaged code base several times faster than in other languages. Keep in mind that unit tests are often not sufficient, make sure that you have several integration / reception tests that you can run quickly after any major changes.

• Use fail fast . Add statements for cases where you feel that your code may be vulnerable.

• You have standard log / error handling that helps you quickly get to the problem.

• Use the IDE (pyDev works for me), which provides the type ahead, the pyLint / Checker integration, which helps to immediately identify common typos and promote some coding standards.

• Pay attention to your import, never import x or import without using.

• Refactoring, a regex search / replace tool is all you need to move methods / refactor a class type.

Answer 6

Incompatible changes in the method signature. This is not as much in Python as it is in Java and C ++.

Python has optional arguments, default values, and much more flexibility in defining method signatures. In addition, duck printing means that, for example, you do not need to switch from any class to an interface as part of a major software change. It's just not that complicated.

How do you find all the places where this method is called? grep works for dynamic languages. If you need to know every place where the method is used, grep (or an equivalent search with IDE support) works fine.

How do you know what operations an instance provides since you do not have a static type to search for?

but. Look at the source. You have no problem with Java / C ++ for object libraries and jar files. You do not need all the necessary tools and tools necessary for these languages.

b. The IDE can provide signature information in many common circumstances. You can easily defeat your arguments for the IDE. When this happens, you should probably consider what you are doing to make sure that it makes sense. If your IDE cannot determine your type information, it may be too dynamic.

from. In Python, you often work through an interactive interpreter. Unlike Java and C ++, you can explore your instances directly and interactively. You do not need a complex development environment.

Example:

  >>> x= SomeClass() >>> dir(x) 

How do you handle / prevent input errors? Same as static languages: you do not prevent them. You find and correct them. Java can only find a specific class of typos. If you have two similar class or variable names, you may run into serious problems even when checking for a static type.

Example:

 class MyClass { } class MyClassx extends MyClass { } 

A typo with these two class names can lead to chaos. “But I wouldn’t take this position using Java,” people say. Agreed. I also did not put myself in this position with Python; you make classes that are very different from each other and will work early if they are used incorrectly.]

Is UnitTest used as a replacement for checking static type? Here's another point of view: static type checking is a replacement for a simple and straightforward design.

I worked with programmers who were not sure why the application was working. They could not understand why things were not going; he did not know the difference between an abstract superclass and an interface, and could not understand why a change in location makes a bunch of other modules in a separate JAR file failure. The static type test gave them false confidence in the wrong design.

Dynamic languages ​​allow programs to be simple. Simplicity is a replacement for static type checking. Clarity replaces static type checking.

Answer 7

My general rule is to use dynamic languages ​​for small non-critical projects and statically typed languages ​​for large projects. I find that code written in a dynamic language like python, for example, gets tangled up faster. This is partly due to the fact that it is much faster to write code in a dynamic language, which leads to shortcuts and worse design, at least in my case. This is partly because I have IntelliJ for quick and easy refactoring when I use Java, which I don't have for python.

Answer 8

The usual answer to this is testing testing testing. It is assumed that you have an extensive set of unit test and run it often, especially before the new version is available on the Internet.

Proponents of dynamically typed languages ​​do what you need to check anyway, because even with static typed compliance with the rough rules of the type system, only a small fraction of what can go wrong is distributed.